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SUMMARY 


Air  Force  enlisted  training  is  an  extensive  and  expensive  operation. 
Almost  all  of  the  thousands  of  individuals  who  enter  the  Air  Force  each 
year  undergo  some  type  of  technical  training  for  their  Air  Force  specialty 
(AFS),  and  most  receive  some  type  of  advanced  technical  and  military 
training  at  various  points  in  their  careers.  Such  training  is  provided  in 
a  number  of  different  settings;  e.g.,  in  resident  technical  courses,  in 
Field  Training  Detachments  (FTDs),  via  correspondence  courses,  or  through 
on-the-job  training  (OJT).  Decisions  which  influence  training  outcomes 
are  made  by  a  number  of  Air  Force  agencies  responsible  for  enlisted 
personnel  utilization  and  training.  Currently,  coordination  of  such 
training  decisions  is  primarily  through  AFS-specific  Utilization  and 
Training  Workshops  (U  &  TWs),  but  U  &  TW  decisions  have  tended  to  focus 
primarily  on  initial  skills  training  programs  and  Specialty  Training 
Standards  (STSs).  At  the  request  of  HQ  USAF  and  Air  Training  Command,  the 
Training  Decisions  System  (TDS)  is  being  developed  as  a  computer-assisted 
training  decision  aid  to  assist  Air  Force  managers  in  defining  the  jobs 
and  training  requirements  of  an  AFS,  in  assessing  the  possible  effects  of 
changing  jobs  or  training  programs,  and  in  making  better  training 
decisions. 

The  TDS  is  composed  of  four  subsystems.  The  Task  Characteristics 
Subsystem  (TCS)  identifies  groups  of  related  tasks  and  how  they  are  (or 
might  be)  allocated  to  various  training  settings  to  achieve  the  required 
proficiency.  The  Field  Utilization  Subsystem  (FUS)  models  the  jobs  and 
training  programs  of  the  AFS  as  a  basis  for  understanding  training 
requirements,  and  develops  ideas  for  alternative  approaches.  The 
Resou rce/ Cost  Subsystem  (RCS)  develops  estimates  of  AFS  training  resources 
and  costs,  and  assesses  the  capacity  of  units  to  provide  such  training  for 
current  and  alternative  utilization  and  training  patterns.  The 
Integration/Optinization  Subsystem  (IOS)  permits  managers  to  examine  the 
possible  consequences  of  their  training  decisions  in  terms  of  costs  or 
meeting  required  proficiencies;  it  will  also  aid  managers  in  optimizing 
such  training  decisions. 

The  TCS  has  two  components.  The  Task  Training  Module  (TTM) 
Construction  component  was  developed  by  assessing  low-technology,  medium- 
technology,  and  high-technology  task  clustering  methodologies.  A 
combination  of  medium-  and  high-technology  approaches  was  selected  as  an 
efficient  means  of  developing  TTMs.  The  recommended  three-step  process 
involves  (a)  task  clustering  using  Comprehensive  Occupational  Data 
Analysis  Programs  (CODAP),  (b)  preliminary  interpretation  by  an  analyst, 
and  (c)  having  subject-matter  experts  (SMEs)  name  and  refine  the  task 
dusters  as  TTMs.  The  TTM  Allocation  component  collects  SME  [judgments  on 
how  much  proficiency  can  be  achieved  for  each  TTM  in  various  training 
settings,  and  uses  this  information  to  estimate  TTM  proficiency-by-setting 
functions.  Three  measures  of  proficiency  were  pilot-tested  with  two  Air 
Force  specialties  (AFs  riiyy,  Security  Police;  AFS  328X4.  Avionic  Inertial 
arid  Radar  Navigation  Systems),  and  the  final  procedure  validated  with  two 
additional  AFSs  (AFS  423X1,  Aircraft  Environmental  Systems;  AFS  305X4, 
Electronic  Computer  and  Switching  Systems).  Detailed  information  for 
applying  the  procedures  developed  for  both  components  is  presented. 
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PREFACE 


The  Task  Characteristics  Subsystem  (TCS)  is  one  of  four  subsystems  of 
the  Training  Decisions  System  (TDS)  research  and  development  (R  &  D) 
effort-  The  TDS  is  a  multi-year  R  &  D  effort  consisting  of  four  major 
research  tasks  and  is  sponsored  by  HQ  USAF/DPPT  and  HQ  ATC/XPC.  It  is 
being  accomplished  under  Project  7734  and  is  being  developed  to  provide  a 
more  systematic  and  integrated  approach  to  training  management  decisions. 
For  a  general  overview  of  the  system,  see  AFHRL-TP-87-25,  Training 
Decisions  System:  Overview,  Design,  and  Data  Requirements.  For  additional 
details,  see  the  reports  cited  in  the  Reference  List  for  this  report. 

A  project  Qf  this  magnitude  requires  the  cooperation  and  dedication  of 
many  people.  In  this  respect,  we  acknowledge  Mr.  Wayne  Archer  and  Mr. 
Bill  Phalen  of  the  Manpower  and  Personnel  Division  for  sharing  their 
knowledge  of  Comprehensive  Occupational  Data  Analysis  Programs  (CO DAP)  and 
expediting  the  development  of  new  ASCII  CODAP  task  clustering  procedures. 
Mrs.  Connie  Villarreal  of  the  Information  Sciences  Division  provided 
advice  and  expedient  programming  support  when  needed.  Special  thanks  go 
to  all  the  Major  Commands  and  functional  staff  offices  which  provided 
assistance  by  making  available  the  subject-matter  experts  who  were 
instrumental  in  reviewing  the  TCS  data  for  the  AFSs  involved  in  this 
effort. 
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1.0  INTRODUCTION 


The  Task  Characteristics  Subsystem  (TCS)  is  one  of  four  basic  sub¬ 
systems  of  the  Training  Decisions  System  (TDS).  Its  functions  within  the 
TDS  are  (a)  to  derive  groups  of  Occupational  Survey  (OS)  tasks  for  an  Air 
Force  specialty  (AFS)  which  should  be  trained  together,  (b)  to  specify  the 
setting (s)  in  which  these  groups  of  tasks  can  be  trained,  and  (c)  to 
estimate  the  time  required  to  train  the  tasks  in  the  settings  and  Air 
Force  managers'  preferences  for  this  allocation  of  training  among  the 
training  settings.  This  report  documents  the  research  and  development  of 
the  procedures,  software,  and  instrumentation  of  the  TCS  to  support  these 
objectives. 


1.1  Air  Force  Training  Decision-Making 

Training  decision-making  is  a  process  of  balancing,  either  implicitly 
or  explicitly,  a  number  of  separate,  and  possibly  inconsistent,  con¬ 
siderations.  From  an  instructional  viewpoint,  material  to  be  trained 
should  be  matched  as  closely  as  possible  to  the  instructional  madia  to  be 
used.  Some  types  of  content  may  be  imparted  through  trainee  self-study; 
other  topics  may  require  closer  supervision  and  feedback;  still  other 
skills  may  be  trained  most  effectively  through  hands-on  experience  and 
extensive  practice.  From  the  viewpoint  of  the  effective  utilization  of 
manpower,  training  should  be  structured  to  provide  adequate  numbers  of 
fully  trained  personnel  where  and  when  required.  Changes  in  the 
deployment  of  personnel,  technology,  and  situational  demands  affect  the 
way  work  is  structured  and  so,  patterns  of  training.  Finally,  from  a 
financial  perspective,  the  personnel  and  resource  requirements  for 
providing  training  may  differ  substantially  from  one  type  of  training  to 
another.  All  other  factors  being  equal,  the  least  expensive  training 
forum  is  to  be  preferred. 

Ideally,  the  instructional  setting  best  suited  for  the  presentation  of 
a  particular  skill  or  knowledge  is  also  the  least  expensive  and  is  capable 
of  providing  sufficient  numbers  of  fully  trained  personnel  where  and  when 
they  are  required.  In  practice,  instructional,  manpower  and  personnel 
utilization,  and  financial  considerations  must  be  balanced  in  deciding  who 
gets  trained,  when,  where,  and  on  what  skills  and  knowledges. 

Several  Air  Force  agencies  are  currently  responsible  for  managing 
personnel  and  training;  consequently,  the  present  system  for  planning  and 
decision-making  is  very  decentralized  (Mitchell,  Sturdevant,  Vaughan,  & 
Rueter,  1987).  Different  managers  in  Air  Training  Command  (ATC)  have 
direct  responsibility  for  resident  classroom  training,  Field  Training 
Detachments  (FTDs),  mobile  training  teams,  and  Career  Development  Courses 
(CDCs).  The  Military  Personnel  Center  is  the  Air  Force  manager  for  on- 
the-job  training  (OJT),  and  ultimately,  every  supervisor  has  OJT 
responsibilities.  Manpower,  personnel,  and  functional  policies  often  have 
training  implications.  For  example,  RIVET  WORKFORCE  is  a  major  initiative 
to  restructure  AFSs  to  better  meet  functional  requirements.  However,  such, 
occupational  restructuring  has  major  implications  for  training. 


Systematic  efforts  are  being  made  to  coordinate  the  efforts  of  these 
groups.  For  example,  Specialty  Training  Standards  (STSs)  serve  as 
contracts  between  AT C  and  the  major  commands  concerning  what  training  will 
be  provided  in  formal  resident  courses  and  what  will  be  provided 
elsewhere.  Also,  Utilization  and  Training  Workshops  have  been  used  to 
bring  together  Air  Staff  and  major  command  function  ail  managers,  subject- 
matter  experts  (SMEs)  from  base-level  units,  Military  Personnel  Center 
classification  staff,  and  Air  Force  training  managers,  so  that  coordinated 
decisions  can  be  achieved.  This  type  of  conference  provides  an 
interactive  decision  capability  where  all  viewpoints  concerning  a 
specialty  can  be  considered. 

Based  partially  on  the  success  of  the  Utilization  and  Training 
Workshops,  Training  Planning  Teams  have  recently  emerged  in  the  area  of 
training  requirements  specification  and  training  development.  The  primary 
responsibilities  of  these  groups  reside  in  anticipating  the  training 
requirements  for  each  stage  in  the  life  cycle  of  a  weapon  or  support 
system  and  in  Air  Force  career  qualification  (see  AFR  35-8).  Training 
Planning  Team  efforts  are  documented  in  a  Training  Development  Plan,  which 
serves  to  guide  the  development  of  training  for  a  career  field  (or  new 
weapon  system)  for  several  years  into  the  future.  Beyond  the  development 
of  a  formal  plan,  the  team  is  also  responsible  for  seeing  the  plan 
implemented,  meeting  periodically  to  review  progress  and  determine  if 
further  modifications  of  the  plan  are  required. 

The  effectiveness  of  these  activities  is  limited,  however,  as  the 
information  available  to  the  decision-makers  is  often  fragmentary  and 
nonsystematic.  The  current  pattern  of  personnel  utilization  is  described 
by  the  OS  program.  Expertise  regarding  the  appropriateness  of  training  a 
given  topic  in  a  particular  setting  resides  with  training  and  functional 
managers.  However,  given  limited  personnel  and  resource  capacities  for 
individual  training  settings,  the  tradeoffs  of  training  different  skills 
in  particular  settings  are  generally  unclear,  due  to  the  number  of 
distinct  skills  and  knowledges  n  an  AFS  and  the  number  of  possible 
training  settings.  The  associated  costs  of  providing  training  in  given 
settings  are  also  unknown,  as  well  as  the  implications  of  alternative 
patterns  of  personnel  utilization.  Finally,  these  groups  of  Air  Force 
managers,  with  different  backgrounds  and  perspectives,  often  lack  a  common 
vocabulary  or  way  of  looking  at  the  issues  and  potential  courses  of 
action. 

The  TDS  and  its  subsystems  will  be  a  means  of  bringing  together 
information  of  various  types — instructional,  personnel  utilization,  and 
financial--in  a  common  framework  to  aid  Air  Force  training 
decision-makers.  It  will  provide  the  means  to  model  both  the  current 
training  system  and  alternative  patterns  of  job  structure  and  training. 
It  will  assess  the  implications  of  different  manpower  and  personnel 
practices  and  indicate  the  optimal  training  pattern  given  certain  types  of 
constraints.  In  short,  it  will  assist  training  decision-makers  by 
providing  a  common  vocabulary  and  integrated  information  concerning  the 
instructional,  personnel  utilization,  and  financial  implications  of 
various  personnel  and  training  policies. 
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1.2 


Training  Decisions  System  Subsystem  Integration 


In  addition  to  the  TCS,  which  encompasses  instructional  concerns,  the 
TDS  includes  the  following  subsystems:  the  Field  Utilization  Subsystem 
(FUS),  the  Resource/Cost  Subsystem  (RCS),  and  the  Integration  and 
Optimization  Subsystem  (IOS).  In  general,  the  purpose  of  the  FUS  is  to 
represent  personnel  utilization  factors.  These  include  descriptions  of 
the  current  and  alternative  patterns  of  personnel  utilization  and  training 
(U  &  T)  and  Air  Force  managers'  preferences  for  these  different  patterns. 
The  RCS  will  be  designed  to  estimate  the  costs,  resource  requirements,  and 
resource  capacities  of  different  training  sites--the  financial 
considerations  involved  in  training  decision-making.  Finally,  the  IOS 
will  integrate  the  products  from  each  of  these  subsystems,  so  that  the  TDS 
can  model  or  optimize  characteristics  of  the  personnel  training  system. 

The  TCS  produces  twc  major  products  that  will  be  used  in  the  modeling 
and  optimization  of  personnel  use  and  training.  The  first  are  Task 
Training  Modules  (TTMs) .  TTMs  are  groups  of  OS  tasks  for  which  it  is 
advantageous  to  train  them  together  (the  exact  criteria  used  to  identify 
TTMs  will  be  discussed  in  the  next  section  of  this  report).  TTMs  are  the 
basic  units  of  analysis  used  in  the  TDS.  The  FUS  will  describe  personnel 
use  (i-e.,  jobs)  and  training  in  terms  of  the  TTMs  involved.  The  RCS  will 
use  TTMs  as  the  units  for  estimating  training  costs  and  resource 
capacities. 

The  second  TCS  product  is  a  description  of  how  training  on  TTMs  can  be 
allocated  among  various  training  settings,  including  the  classroom,  CDCs, 
FTDs,  and  OJT  (OJT,  for  the  purposes  of  this  study,  is  all  on-the-job 
experience  that  is  required  in  order  to  reach  minimum  performance 
standards  for  the  TTMs;  it  is  not  limited  to  the  formal  upgrade  OJT 
program).  This  TTM  allocation  description  includes  the  number  of  hours  of 
training  in  each  setting  that  are  required,  on  average,  to  reach  minimum 
performance  standards  for  each  TTM  and  Air  Force  managers'  preferences  for 
this  allocation  of  training.  Training  times  are  used  by  the  RCS  in 
establishing  costs  and  resource  capacities.  Preferences,  on  the  other 
hand,  are  outcomes  from  the  TCS  that  could  be  reported  for  the  modeling  or 
optimization  of  personnel  use  and  training  patterns. 

Their  proposed  uses  place  certain  requirements  on  the  TCS  products. 
With  regard  to  TTMs,  tnese  task  groups  should  be  conducive  to  the 
description  of  the  current  and  alternative  patterns  of  personnel  training 
and  job  structure  in  the  FUS.  However,  since  the  relationship  between  OS 
tasks  and  jobs/training  is  complex  and  overlapping,  the  match  between  TTMs 
and  U  &  T  patterns  will  necessarily  be  approximate.  In  addition,  TTMs 
should  be  relevant  to  a  range  of  different  U  &  T  patterns,  of  which  the 
current  pattern  is  only  one. 

TTMs  should  also  represent  groups  of  OS  tasks  which  are  relatively 
homogeneous  with  respect  to  underlying  skills  and  knowledges  and  which  are 
relatively  distinct  from  other  groups  of  OS  tasks.  From  an  instructional 
viewpoint,  these  are  tasks  which  can  be  trained  in  the  same  setting  (s). 
From  the  perspective  of  personnel  utilization,  TTMs  of  this  type  can  be 
used  to  describe  current  or  alternative  patterns.  And  finally,  from  the 
financial  viewpoint,  such  TTMs  should  encompass  most  of  the  economies  of 
training  that  result  from  common  training  materials,  content,  equipment, 
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and  the  like.  Consequently,  the  associated  costs  of  training  are  additive 
across  TTMs  for  many  different  personnel  and  training  policies. 

Use  of  TTM  allocations  by  the  other  TDS  subsystems  also  places 
requirements  on  these  descriptions.  Descriptions  of  TTM  training 
allocations  should  represent  a  broad  range  of  the  feasible  ways  of 
apportioning  training  among  settings.  This  requirement  permits  the  TDS 
flexibility  in  modeling  and  optimizing  patterns  of  personnel  training. 
The  primary  restriction  on  each  TTM  allocation  is  that  the  training 
provided  in  all  of  the  settings  be  sufficient  to  permit  the  average  airman 
to  perform  the  tasks  in  the  TTM  to  minimum  standards;  that  is,  the  TCS  is 
responsible  for  setting  the  lower  bound  for  training  times  for  modeling  or 
optimization.  Given  that  the  times  are  adequate,  on  average,  the  RCS  can 
use  the  allocations  to  estimate  training  cost. 

The  TCS  is  divided  into  two  components  according  to  these  two  primary 
products  --  the  TTM  construction  component  and  the  TTM  allocation 
component.  Research  and  development  (R  &  D)  of  these  components  is 
described  in  the  next  two  sections  of  this  report.  The  work  discussed  in 
these  two  sections  is  based  on  R  &  D  in  four  AFSs--811XX,  Security,  Law 
Enforcement,  and  Law  Enforcement-Military  Working  Dog  Qualified  Career 
Ladders;  328X4,  Avionic  Inertial  and  Radar  Navigation  Systems  Career 
Ladder;  305X4,  Electronic  Computer  and  Switching  Systems  Career  Ladder; 
and  423X1,  Aircraft  Environmental  Systems  Career  Ladder.  In  the  final 
section,  directions  for  future  research  in  TCS  methodologies  are 
discussed. 


2.0  TASK  TRAINING  MODULE  CONSTRUCTION 

The  requirements  that  TTMs  be  appropriate  for  describing  patterns  cf 
personnel  utilization  and  training  (both  resident  and  field  training),  and 
that  they  encompass  economies  of  training  tasks  together,  imply  two 
general  characteristics  of  TTMs:  They  should  be  composed  of  tasks  that 
share  skill  and  knowledge  requirements,  and  they  should  tend  to  be 
performed  by  the  same  personnel  in  the  field  (although,  not  necessarily  at 
the  same  time).  The  latter  characteristic,  co-performance,  captures 
efficiencies  of  training  tasks  in  the  field,  for  example,  in  FTDs  or 
through  CXJT;  and  these  same  efficiencies  may  be  important  for  resident 
training  and  CDCs.  The  characteristic  of  common  skill  and  knowledges 
assures  that  instruction  on  sets  of  similar  tasks  in  any  setting  can  be 
accurately  represented,  so  that  economies  such  as  those  resulting  from 
shared  resources  or  transfer  of  learning  are  taken  into  account.  To  the 
extent  that  the  characteristics  of  homogeneity  of  skills  and  knowledges 
and  co-performance  are  met,  the  TTMs  formed  should  be  meaningful  to  TDS 
users  and  should  be  useful  for  training  decision-making. 

Two  TTM  construction  methods  were  field-tested  in  AFS  811XX  and  AFS 
328X4 — statistically  clustering  OS  tasks  based  on  co-performance  and  SME 
task  sorting  (several  similarity  measures  were  evaluated  in  addition  to 
co-performance,  and  techniques  other  than  statistical  clustering  and  task 
sorting  were  evaluated;  see  Perrin,  Vaughan,  Yadrick,  Mitchell,  &  Knight, 
1986,  for  more  details).  These  TTM  construction  methods  were  selected  for 
field  test  because  of  their  sensitivity  to  co-performance  and  to  skill  and 
knowledge  homogeneity  requirements.  The  field  test  of  these  methods  is 
discussed  in  the  section  that  follows.  A  more  complete  account  of  the 


R  &  D  of  the  TTM  construction  methods  is  available  in  Perrin,  Vaughan  et 
al.  (1986).  From  the  results  of  those  field  tests,  a  procedure  based  on 
these  techniques  was  developed  and  applied  to  AFS  423X1  and  AFS  305X4.  In 
the  second  section,  this  recommended  procedure  is  described,  along  with 
the  results  of  applying  it  to  the  two  new  AFSs.  Perrin,  Mitchell,  and 
Knight  (1986)  provide  a  more  thorough  description  of  this  work. 
Guidelines  for  using  the  recommended  procedure  are  detailed  in  Appendix  A. 

2.1  Development  of  TTM  Construction  Methods 
2.1.1  Statistically  Clustering  OS  Tasks 

Statistical  clustering  has  a  long  history  in  military  occupational 
analysis.  For  more  than  30  years,  Air  Force  job  analysts  have  used  case 
(person)  cluster  diagrams  to  assist  in  identifying  jobs--groups  of 
personnel  performing  similar  sets  of  tasks.  Over  that  period  of  time, 
standardized  data  collection  instruments  (OS  task  inventories)  have  been 
developed  and  refined,  computing  algorithms  and  diagnostic  statistics  have 
been  devised  and  tested,  important  background  characteristics  have  been 
identified  and  incorporated  into  the  analysis  of  the  structure  of  work, 
and  an  extensive  body  of  research  and  application  has  accrued  (Christal, 
1974;  Mitchell,  Ruck,  &  Driskill,  1988).  By  comparison,  the  notion  of 
statistically  clustering  tasks  to  support  training  decision-making  is 
relatively  new.  Nonetheless,  existing  OS  data  and  Comprehensive 
Occupational  Data  Analysis  Programs  (CODAP)  algorithms  proved  suitable  for 
testing  this  approach  for  TTM  construction. 

The  statistical  clustering  algorithm  that  has  been  applied  to  job 
typing  and  is  utilized  in  CODAP  is  the  average  linkage  clustering 
procedure  (Ward,  1963).  This  procedure  has  performed  well  in  empirical 
studies  that  have  compared  various  clustering  algorithms  (Milligan,  1981; 
Mojena,  1977),  and  consequently,  it  was  selected  as  the  algorithm  to  use 
for  task  clustering.  Similarity  indices  that  were  field-tested  were  of 
two  general  types — co-performance  similarity  and  similarity  based  on  task 
characteristics.  Jn  general,  task  clusters  based  on  the  co-performance 
similarity  measure  more  nearly  met  the  requirements  for  skill/knowledge 
homogeneity  and  co-performance  than  the  clusters  based  on  the  other 
similarity  measures;  consequently,  only  the  research  on  co-performance  is 
discussed  in  this  report. 

TTMs  based  on  OS  task  co-performance  clustering  tend  to  produce  groups 
of  tasks  that  are  performed  together;  that  is,  if  an  airman  performs  one 
task  in  the  group,  it  is  likely  that  he  or  she  also  performs  others  in  the 
same  group  as  part  of  his  or  her  job.  Figure  1  illustrates  the  relation¬ 
ship  between  case  (or  person)  clustering  (the  usual  application  of  CODAP 
to  job  typing)  and  task  clustering  (the  application  to  identify  tasks  that 
are  performed  by  the  same  personnel).  (Note  that  job  typing  normally  uses 
a  relative  time  spent  measure  for  clustering  cases  rather  than  the 
performed/not  performed  dichotomy  depicted  in  Figure  1;  in  normal  job 
typing,  a  number  between  0  and  1  representing  relative  time  spent  on  a 
task  would  replace  the  1's  in  the  figure.)  While  job  typing  involves 
grouping  persons  who  perform  the  same  (or  similar)  sets  of  tasks,  task 
clustering  produces  sets  of  tasks  which  are  frequently  co-performed. 


r 


JOB  TYPING:  CLUSTERING  PERSONS 
WHO  PERFORM  THE  SAME  (OR 
SIMILAR)  SETS  OF  TASKS 

TASKS 

123456789 
1111 
2  111 

3  111111 

4  111111 

5  111 

6  111 

7  111 

8  111 


TTM  CONSTRUCTION:  CLUSTERING  TASKS 
THAT  ARE  PERFORMED  TOGETHER 
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Figure  1.  The  Relationship  Between  Case  Clustering  for  Job  Typing  and 
Task  Clustering  for  TTM  Construction. 
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2.1.2  SME  Task  Sorting 


Perhaps  one  of  the  most  straightforward  ways  to  classify  information 
is  simply  to  have  people  sort  examples  into  categories.  One  such  approach 
is  to  print  examples  on  cards  and  have  people  sort  these  cards;  the  piles 
that  result  constitute  the  categories.  This  procedure  has  a  long  history 
in  psychology  for  research  in  cognitive  modeling,  and  seemed  suitable  for 
use  in  forming  TTMs  from  OS  tasks. 

To  make  the  job  of  sorting  all  of  the  OS  tasks  of  a  specialty  more 
manageable,  some  initial  structure  in  the  form  of  starter  piles  was 
provided.  These  initial  piles  were  not  to  limit  the  sorting  of  tasks  in 
any  way,  but  merely  to  provide  groups  of  manageable  size  based  on  reason¬ 
able  divisions  of  the  subject  matter.  Two  types  of  starter  piles  were 
used  in  the  field  tests:  one  based  on  STS  paragraph  references  and  a 
second  based  on  co-performance  clustering  (as  described  previously) . 

In  addition  to  the  initial  structuring  of  the  task,  the  background  of 
the  SMEs  might  influence  the  groupings  produced.  Specifically,  resident 
technical  trainers  and  field  personnel  might  have  different  points-of-view 
regarding  the  similarity  of  tasks  for  training  purposes.  Consequently, 
separate  groups  of  SMEs,  one  composed  of  technical  trainers  and  one  of 
field  personnel,  were  used  to  sort  each  set  of  starter  piles.  The  four 
groups  involved  in  the  field  tests  can  be  identified  by  their  type  of 
starter  pile  and  their  background  as  follows:  STS/ school;  STS/field; 
co-performance/school;  and  co-performance/field.  The  TTMs  produced  in 
four  independent  card  sortings  permit  pairwise  comparisons  between 
different  initial  structures  and  groups  of  SMEs  with  different  training 
perspectives. 

The  groups  of  SMEs  were  directed  to  rearrange  the  cards  into  piles 
that  represented  groups  of  tasks  that  should  be  trained  together.  The 
directions  to  the  SMEs  stressed  the  importance  of  using  their  expertise  in 
forming  the  TTMs,  and  the  instructions  specifically  noted  that  the 
resulting  piles  might  include  duplicate  tasks  and  be  of  any  size, 
including  single  tasks.  The  groups  of  SMEs  worked  independently  to  sort 
the  cards  into  TTMs,  proceeding  at  their  own  pace  and  using  their  own 
strategies  to  complete  the  task. 

When  each  group  was  satisfied  with  their  task  groupings,  two  groups 
met  to  reconcile  any  differences  between  their  groupings.  This 
reconciliation  phase  allowed  the  groups  to  compare  strategies  used  to  form 
the  TTMs,  decide  upon  the  best  criteria,  and  produce  a  final  set  of  task 
clusters.  The  two  sets  of  reconciliation  sorts  provided  independent 
replication  of  the  results. 

In  AFS  811XX,  all  four  groups  sorted  the  tasks  into  TTMs.  Then,  the 
co-performance/field  and  co- performance/school  groups  met  to  create  one 
reconciliation  sort,  while  the  STS/field  and  STS/ school  groups  formed  the 
other  reconciliation  sort.  Attempts  to  convene  a  group  of  field  SMEs  for 
AFS  328X4  were  unsuccessful,  and  so,  only  two  groups  were  formed: 
STS/school  and  co- performance/ school.  These  two  groups  then  met  to  form  a 
reconciliation  sort  in  AFS  328X4. 
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2.1.3  TTM  Construction  Field  Test  Results 


Consensus  judgments  of  sizable  groups  of  SMEs  concerning  the  tasks  in 
their  specialty  that  should  be  trained  together  are  perhaps  the  best 
criteria  for  TTM  construction  methods.  This  approach,  since  it  was 
designed  to  facilitate  maximum  expression  of  the  participants'  expertise 
in  grouping  tasks,  is  both  a  TTM  construction  method  and  a  standard 
against  which  other  construction  methods  can  be  compared.  Additionally, 
however,  TTMs  identified  from  a  task  co-performance  cluster  diagram 
represent  an  important  criterion — co-performance.  Fortunately,  results 
from  the  field  tests  in  AFS  811XX  and  AFS  328X4  indicate  that  these 
approaches  to  TTM  construction  are  consistent  and  complementary. 

The  Fowlkes  and  Mallows  (1983)  statistic  was  used  to  indicate  the 
consistency  of  task  grouping  results.  This  statistic  ranges  from  0,  when 
no  two  tasks  were  grouped  together  in  both  of  the  two  solutions  being 
compared,  to  1,  when  the  two  solutions  are  identical.  The  Folkes  and 
Mallows  statistic  will  exceed  0.044  in  only  about  one  percent  of  the  cases 
by  chance  alone,  given  the  number  of  tasks  being  sorted  and  the  number  of 
groups  being  formed. 

The  SME  task  sortings  were  quite  consistent.  The  average  Folkes  and 
Mallows  statistic  for  the  comparisons  among  the  different  811XX  task 
groupings  was  0.264,  and  ranged  from  0.199  to  0.304.  Since  there  were 
only  two  328X4  groups,  there  was  only  one  independent  comparison;  the 
Folkes  and  Mallows  statistic  for  that  comparison  was  0.476.  Clearly, 
there  was  a  core  of  task  groupings  about  which  the  experts  agreed  quite 
closely. 

After  an  analyst  had  identified  TTMs  from  the  task  co-performance 
cluster  diagram,  these  groupings  were  compared  to  the  SME  task  sortings. 
Significantly,  the  overall  level  of  agreement  between  the  SMEs'  card  sorts 
and  the  task  co-performance  clusters  tended  to  be  as  high  or  higher  than 
the  agreement  between  different  groups  of  SMEs.  In  AFS  328X4,  the  average 
level  of  agreement  between  the  co-performance  clusters  and  the  SME 
sortings  (as  indicated  by  the  Folkes  and  Mallows  statistic)  was  0.247. 
For  AFS  811XX,  the  Folkes  and  Mallows  statistic  averaged  0.428  for  the 
same  comparisons.  Taken  together,  these  findings  suggest  that  the 
similarities  in  tasks  that  result  in  frequent  co-occurrence  in  performing 
one's  job  are  important  considerations  in  determining  what  tasks  should  be 
trained  together.  Apparently,  the  co-performance  clusters  captured  the 
core  of  agreement  in  the  SMEs'  judgments.  This  conclusion  must  be 
qualified,  however,  as  a  significant  number  of  tasks  in  both  specialties 
(115  for  811XX  and  35  for  328X4)  could  not  be  classified  by  the  analyst, 
based  solely  on  the  co-performance  cluster  diagram. 

The  type  of  starter  pile  used  in  the  card  sorting  exercise  (co¬ 
performance  or  STS  paragraph)  had  no  appreciable  effect  on  the  results. 
The  background  of  the  SMEs  (school  or  field)  apparently  did  influence  the 
solution.  Field  SMEs'  TTMs  more  closely  matched  the  co-performance 
clusters  than  did  the  school  SMEs'  sortings.  Also  in  AFS  811XX,  the 
results  from  the  two  field  groups  showed  the  highest  level  of  agreement, 
and  the  reconciliation  sorts  were  more  similar  to  the  field  SMEs' 
solutions  than  to  the  school  SMEs'  TTMs. 
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In  addition  to  the  consistency  of  results  obtained  using  different  TTM 
construction  methods,  the  relative  costs  and  flexibility  of  the  approaches 
were  considered.  Task  co-performance  clustering  is  a  relatively  efficient 
and  cost-effective  method  of  contructing  TTMs.  The  basic  routines  for 
task  clustering  have  already  been  implemented  in  a  rewrite  of  CO  DAP  for 
the  Air  Force  (called  ASCII  CODAP  in  the  rest  of  this  paper;  see  Phalen, 
Weissmuller,  &  Staley,  1985,  for  an  overview  of  ASCII  CODAP).  Opera¬ 
tional  costs  would  be  largely  limited  to  computer  charges  for  compiling 
the  task  cluster  diagram  and  an  analyst's  time  in  interpreting  it. 

Unfortunately,  the  clustering  procedure  has  some  limitations  in  its 
flexibility  for  constructing  TTMs.  Specifically,  the  current  clustering 
procedure  does  not  allow  for  the  assignment  of  a  task  to  two  or  more  TTMs, 
a  situation  that  might  be  of  value  in  certain  situations.  Additionally, 
it  is  sometimes  difficult  to  assign  a  task  to  a  group  based  solely  on  a 
cluster  diagram.  (In  job  typing,  it  is  common  to  have  from  10%  to  20%  of 
the  cases  unclassified.)  A  new,  nonhierarchical  clustering  algorithm 
holds  great  promise  for  reducing  or  eliminating  these  difficulties 
(Mitchell  &  Phalen,  1985);  however,  this  procedure  is  not  yet  fully 
operational,  and  so,  complete  evaluation  of  its  usefulness  for  the  TDS 
cannot  yet  be  determined. 

On  the  other  hand,  SME  card  sorting  is  more  flexible  than  co¬ 
performance  clustering.  The  instructions  for  card  sorting  were  written  to 
permit  maximum  flexibility  in  structuring  TTMs.  Emphasis  was  placed  on 
using  one's  expertise  in  forming  modules  of  tasks  that  should  be  trained 
together,  and  the  possibility  of  using  duplicate  tasks  for  this  purpose 
was  specifically  mentioned. 

The  costs  associated  with  SME  card  sorting  can  be  rather  high,  how¬ 
ever.  Results  from  the  field  tests  indicate  that  (a)  multiple  groups  of 
at  least  three  SMEs  (more  if  the  specialty  is  diverse)  should  be  involved, 
and  (b)  at  least  one  group  should  be  composed  of  field  SMEs.  Given  the 
requirement  for  multiple  SME  groups  composed  of  both  school  and  field 
personnel,  and  considering  that  card  sorting  required  a  minimum  of  2  days 
(3  if  a  reconciliation  sort  was  produced),  arranging  for  card  sorting  in 
any  particular  specialty  may  be  difficult  and  costly.  These  difficulties 
will  apparently  be  compounded  for  highly  specialized  career  fields,  when 
personnel  are  dispersed  geographically,  or  when  workload  is  high. 


2.2  Recommended  Procedure 

Ihe  TTM  construction  methods  of  co- performance  clustering  and  SME  card 
sorting  were  found  to  have  both  strengths  and  weaknesses.  The  recommended 
procedure  of  co-performance  clustering,  followed  by  SME  refinement, 
capitalizes  on  the  strengths  of  both  methods,  while  avoiding  many  of  the 
pitfalls.  This  procedure  was  applied  to  TTM  construction  for  AFS  305X4 
and  AFS  423X1.  The  results  described  in  this  section  reflect  the 
validation  work  in  these  two  specialties. 

The  recommended  TTM  construction  procedure  involves  three  steps  as 
follows; 
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1.  statistically  cluster  OS  tasks  for  the  specialty  using 
co-performance  as  the  similarity  measure; 

2.  interpret  the  co-performance  cluster  diagram,  identifying 
initial  TIMs  based  on  patterns  of  between  and  within 
group  homogeneity;  and 

3.  have  SMEs  in  the  specialty  name  the  task  groupings,  place 
tasks  that  could  not  be  grouped  based  on  the  clustering 
results,  and  refine  the  initial  clusters  to  form  final  TIMs. 

The  discussion  of  this  validation  work  is  organized  around  these  three 
steps.  First,  the  method  of  task  co-performance  clustering  is  described. 
Then,  training  for  cluster  diagram  interpretation  and  comparisons  of  three 
independent  interpretations  for  each  specialty's  cluster  diagram  are 
summarized.  Third,  steps  for  SME  review  of  the  initial  TIMs  are 
discussed,  and  these  final  clusters  are  compared  to  the  initial 
groupings.  In  each  of  these  sections,  the  personnel  and  resource 
requirements  for  applying  the  procedure  cure  noted.  In  the  final  section, 
the  internal  consistency  of  TIMs  and  their  distinctiveness,  one  from 
another,  have  been  assessed  by  using  task  analysis. 

2.2.1  Co-performance  Clustering 

The  first  step  in  the  recommended  TIM  construction  procedure  is  to 
statistically  cluster  the  OS  tasks  in  the  specialty.  Task  co-performance 
clustering  is  supported  in  ASCII  OODAP.  The  first  step  in  producing  a 
task  co-performance  cluster  diagram  is  to  transpose  the  case  data  file 
using  a  routine  called  XPObE.  This  routine,  as  the  name  implies,  converts 
the  file  from  a  case-oriented  to  a  task-oriented  form.  Rather  than  task 
performance  data  for  each  respondent,  the  file  is  composed  of  respondent 
performance  data  for  each  task.  A  practical  limitation  is  imposed  by  this 
approach  in  that  the  number  of  cases  effectively  becomes  the  number  of 
tasks  in  the  transposed  file,  and  ASCII  OODAP  is  limited  to  3,000  tasks 
(transposed  cases)  that  can  be  clustered  at  one  time. 

Sample  sizes  for  305X4  and  423X1  were  less  than  this  limit,  however, 
so  XPOSE  was  used  to  transpose  the  case  data  files.  The  XPOSE  control 
cards  specified  the  "raw  task  responses"  transpose  of  the  file.  Then, 
clustering  was  performed  using  the  regular  sequence  of  OVRLAP,  GROUP,  and 
DIAGRM,  specifying  the  co-performance  option  in  the  OVRLAP  routine. 
Finally,  the  OS  tasks  can  be  linked  to  TPATh  (KPATH)  numbers,  so  that  the 
groups  can  be  interpreted  by  studying  the  tasks  in  TPATh  order  (FRTVAR; . 

Since  the  completion  of  this  research,  several  additional  CODAP 
products  that  support  interpretation  of  task  cluster  diagrams  have  been 
developed  by  Texas  Maxima  Corporation  under  contract  with  AFHRL.  Fhalen, 
Staley,  and  Mitchell  (1987)  discuss  seme  of  the  new  routines  that  have 
been  developed  for  this  purpose. 

The  cost  of  this  TIM  construction  step  has  been  estimated  to  be 
approximately  $300  to  $350  per  specialty,  plus  computer  charges. 
Undoubtedly,  this  step  will  become  more  costly  and  time-consuming  as 
additional  TIM-relevant  programs  are  added  to  CODAP;  however,  the  quality 
of  the  resulting  interpretations  should  increase  as  well,  potentially 
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reducing  costs  for  later  TIM  construction  steps. 

2.2.2  Cluster  Diagram  Interpretation 

The  second  step  in  the  recommended  TIM  construction  procedure  is  to 
interpret  the  task  co-performance  cluster  diagram.  For  the  purposes  of 
the  validation  work  in  AFSs  305X4  and  423X1,  two  AFHRL  analysts  were  given 
3  days  of  training  on  this  task.  The  content  of  the  training  was  as 
outlinad  below: 

3.  Orientation  to  Hierarchical  Clustering 

a.  Discussion  of  Air  Force  occupational  analysis 

b.  Examination  of  the  OS  task  inventory  for  APS  423X1 

c.  Walk-through  of  the  AFS  423X1  cluster  diagram 

d.  Examination  of  the  AFS  423X1  OS  Report 

e.  Discussion  of  AFHRL  data  files  (e.g.,  Occupational  Research 

Data  Base  reports,  OS  Reports) 

f.  Discussion  of  the  co-performance  similarity  measure 

g.  Discussion  of  the  new  version  of  COCAP 

2.  Walk-through  of  TTM  Construction  in  AFS  328X4 

a.  Examination  of  new  OODAP  cluster  diagram  format 

b.  Criteria  for  initial  TTM  selection  and  use  of  PRTVAR 

c.  Examination  of  booklets  used  by  SMEs  to  refine  TIMs 

d.  Discussion  of  TTM  review  by  SMEs  in  AFS  328X4 

e.  Second  review  of  cluster  diagram  in  light  of  SME  results 

f.  Final  TTMs  and  their  uses  in  TDS 

3.  Practicum  —  Analysis  of  AFS  423X1  diagram 

a.  Handout  copies  of  AFS  423X1  task  cluster  diagram 

b.  Handout  copies  of  AFS  423X1  FKTVAR 

c.  Discussion  of  criteria  and  approach  to  analysis  (including 

need  for  independent  analyses) 

d.  Analysis  of  diagram  (with  assistance  as  required) 

4.  Comparison  of  Results 

a.  Review  of  exercise  goals  and  status 

b.  Completion  of  TTM  listings 

c.  Comparison  of  outcomes  (in  terms  of  number  and  general  size 

of  initial  TIMs  identified) 

d.  Discussion  of  agreement  and  differences 

Following  training,  the  two  AFHRL  analysts  and  contractor  personnel 
independently  interpreted  the  AFS  423X1  and  AFS  305X4  task  co-performance 
cluster  diagrams.  Although  the  interpretations  differed  somewhat  in  the 
specificity  of  the  initial  TTMs  identified  (some  analysts  favoring  broad 
TTMs  with  many  tasks;  others  preferring  smaller,  specific  TIMs) ,  the 
interpretations  of  the  diagrams  were  highly  consistent.  The  average 
Folkes  and  Mallows  statistic  for  all  comparisons  between  different 
solutions  for  AFSs  423X1  and  305X4  was  0.89. 

Estimates  of  the  cost  of  the  initial  training  for  the  analysts  were 
about  $800  per  trainee.  With  the  limited  number  of  computer  products  that 
were  available  to  support  initial  TTM  identification,  analysis  of  task 
cluster  diagrams  would  probably  vary  from  about  3  to  10  days, 
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depending  on  the  complexity  of  the  specialty.  The  associated  personnel 
costs  would  range  from  about  $380  to  $1,300  per  specialty.  Again,  this 
cost  will  undoubtedly  grow  as  the  TTM  identification  process  is  refined 
and  additional  computer  products  are  developed. 

2.2.3  SME  Refinement 

The  final  step  in  the  recommended  TTM  construction  procedure  is  to 
have  a  representative  group  of  SMEs  review,  name,  and  refine  the  initial 
TTMs.  In  this  step,  care  should  be  taken  to  select  an  adequate  group  of 
SMEs  to  represent  all  important  aspects  of  the  career  field  (weapon 
system,  major  command,  type  of  equipment,  etc.).  Additionally,  earlier 
research  (discussed  in  section  2.1.3)  indicates  that  these  SMEs  should  be 
selected  from  both  training  and  field  personnel,  as  the  perspectives  of 
these  two  groups  may  differ. 

Objectives  of  this  refinement  step  are  threefold.  First,  the  SMEs 
should  name  each  TTM  with  a  title  that,  as  clearly  as  possible, 
communicates  the  content  of  the  module.  This  process  apparently  helps  the 
SMEs  identify  the  common  theme  among  the  tasks  and  so  highlights 
inappropriately  grouped  tasks.  A  clear  TTM  title  will  also  be  invaluable 
for  communicating  content  in  later  surveys  and  for  making  the  final  TDS 
easy  for  users  to  operate.  The  second  objective  of  the  refinement  step  is 
to  add  or  delete  tasks  from  TTMs  to  form  the  most  cohensive  and 
distinctive  groupings  possible.  Finally,  SMEs  should  group  the  tasks  that 
could  not  be  classified  by  the  analyst  based  on  the  co-performance  cluster 
diagram.  This  may  mean  forming  new  TTMs  or  simply  adding  these  tasks  to 
existing  modules. 

Of  special  interest  is  the  423X1  TTM  refinement,  since  this  list  of 
TTMs  was  completely  reviewed  twice,  once  at  Randolph  AFB  and  once  at 
Tinker  AFB.  In  the  first  refinement,  the  SMEs  tended  to  favor  more 
specific  TTMs  than  two  of  the  three  analysts,  forming  73  TTMs.  The  second 
SME  refinement  resulted  in  a  broadening  of  the  TTMs,  as  the  SME  combined 
task  groupings  to  form  58  final  task  modules.  It  is  interesting  to  note 
that  often  the  SMEs'  TTM  divisions  and  combinations  could  be  directly 
linked  to  the  original  cluster  diagram.  Results  from  comparing  the 
initial  TTMs  (analyst's  interpretations  of  the  cluster  diagram)  and  the 
SME  refinements  using  the  Fowlkes  and  Mallows  agreement  statistic 
indicated  that  analysts  tend  to  agree  with  analysts  more  closely  (average 
Fowlkes  and  Mallows  of  0.89)  than  they  agree  with  the  SME  reviews  (average 
Fowlkes  and  Mallows  of  0.58).  Additionally,  the  two  independent  SME 
reviews  in  AFS  423X1,  while  differing  in  level  of  specificity,  agreed 
quite  closely  with  each  other  (Folkes  &  Mallows  of  0.87).  In  short, 
analysts'  interpretations  of  cluster  diagrams  are  consistent  with  but  not 
equivalent  to  SME-formed  TTMs;  thus  SME  review  is  an  important  step  in 
creating  stable,  meaningful  task  groupings. 

Costs  incurred  in  the  SME  refinement  step  would  include  the  salaries 
of  the  SMEs  and  support  personnel  involved.  The  number  of  SMEs  will  vary 
with  the  diversity  of  the  specialty.  Three  to  five  SMEs  appears  suf¬ 
ficient  in  career  fields  with  simple  structures?  twelve  to  fifteen  SMEs 
should  be  sufficient  for  all  except  perhaps  the  most  diverse  specialties. 
For  a  complete  review,  especially  if  this  is  the  first  and/or  only  review, 
1  day  of  SME  time  should  be  allotted.  Given  these  factors,  personnel 
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costs  for  refinement  might  range  from  roughly  $350  to  $1,7  00,  again 
depending  on  the  complexity  of  the  specialty.  Travel  and  housing 
expenses,  if  required,  would  have  to  be  added  to  this  figure  to  arrive  at 
a  total  cost  for  TTM  refinement. 

Using  figures  cited  earlier  for  task  clustering,  diagram  inter¬ 
pretation,  and  SME  refinement,  the  total  costs  for  TTM  construction  should 
range  from  about  $1,030  in  simple  AFSs  to  $3,300+  for  a  very  complex 
specialty.  These  figures  do  not  include  initial  analyst  training  cost 
(see  section  2.2.2),  nor  do  they  include  computer  charges  (other  than 
programmer  time)  or  SME  and/or  TDS  personnel  travel  and  housing  expenses. 


2.3  Task  Analysis 

An  important  requirement  of  TTMs  is  that  they  should  represent 
relatively  independent  and  distinct  areas  of  subject-matter  knowledge  and 
procedural  skills,  so  that  TTM  training  costs  will  be  additive.  An 
additional  importar.c  characteristic  is  that  they  contain  related  tasks,  so 
that  training  content  can  be  clearly  and  unambiguously  communicated  during 
data  collection  and  to  users  of  the  final  system.  A  test  that  reflects 
both  the  distinctness  and  internal  consistency  of  TTMs  is  task  analysis. 
The  requisite  knowledges  and  procedural  skills,  as  indicated  through  task 
analysis,  should  be  more  similar  for  tasks  within  a  TTM  than  for  tasks 
from  different  TTMs.  Overall,  task  analysis  indicated  that  the  tasks 
within  a  TTM  were  more  similar  than  tasks  from  different  TTMs,  further 
supporting  the  validity  of  the  recommended  TTM  construction  procedure. 

Two  sources  of  information  guided  the  task  analysis  work:  the  Task 
Analysis  Handbook  (AFHRL-TR-79-45;  see  Eschenb renner,  DeVries,  Miller,  & 
Ruck,  1980,  and  DeVries,  Eschenbrenner  &  Ruck,  1980),  and  the  draft 
Training  Development  Service  (USAFOMC/OMT)  Procedures  Guide,  Chapter  8.0, 
Task  Analysis. 

Four  SMEs  were  interviewed  at  two  separate  locations  for  each 
specialty.  All  SMEs  interviewed  were  extremely  well  qualified  in  their 
respective  career  fields.  Four  TTMs  were  selected  by  AFHRL  personnel  in 
each  specialty  for  task  analysis.  However,  only  three  of  these  were 
analyzed  by  the  423X1  SMEs,  due  to  time  constraints.  The  procedure 
followed  for  each  TTM  was  to  present  the  SME  with  an  individual  task  from 
the  TTM,  and  then  have  that  SME  complete  a  four-part,  five-page  task 
analysis  worksheet  developed  by  the  USAF  Occupational  Measurement  Center. 

The  task  analyses  of  the  tasks  within  TTMs  indicated  that  the  TTMs 
tend  to  be  both  homogeneous  with  respect  to  the  requisite  skills  and 
knowledges  and  distinct  from  other  TTMs.  For  several  TTMs,  the  equipment 
required,  the  environment,  and  the  references  were  all  very  similar  or  the 
same  for  all  of  the  tasks  in  the  TTM.  By  contrast,  some  of  the  TTMs 
selected  for  task  analysis  included  preventive  maintenance  inspection 
(PMI)  tasks,  including  one  TTM  which  contained  only  one  task.  Such  tasks, 
by  their  very  nature,  require  a  variety  of  skills  and  knowledges, 
especially  when  an  entire  system  or  subsystem  of  equipment  or  test 
equipment  is  involved.  The  question  of  whether  these  PMI  tasks  should 
stand  alone  or  be  incorporated  into  the  TTM  for  the  system/subsystem  is^ 
one  of  the  specificity  of  the  module — an  issj^e  on  which,  as  was  noted* 
previously,  even  the  experts  do  not  always  agree. 
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2.4  Conclusions 


The  primary  issue  raised  by  this  validation  work  is  not  one  of  the 
adequacy  of  the  TTM  construction  procedure  itself;  rather,  the  primary 
issue  is  one  of  the  appropriate  level  of  specificity  for  TTMs.  In 
interpreting  the  task  co-performance  cluster  diagiam,  it  was  a  question  of 
'how  far  down  the  diagram'  one  chooses  initial  TTMs.  In  the  refinement 
step,  SMEs  were  found  to  differ  in  the  degree  of  specificity  they  believed 
was  appropriate.  In  task  analysis,  the  problem  was  encountered  in  the 
form  of  the  placement  of  PMI-type  tasks.  Furthermore,  this  problem  seems 
to  be  more  severe  for  management  and  supervisory  tasks  than  for  technical 
tasks;  that  is,  there  appears  to  be  less  agreement  as  to  the  appropriate 
level  of  specificity  for  TTMs  that  cover  management/supervisory  tasks 
compared  to  technical  tasks.  But,  as  has  been  noted  elsewhere  (Perrin, 
Vaughan  et  al.,  1986),  choice  of  a  level  of  specificity  is  strongly 
influenced  by  the  ultimate  uses  for  the  TDS.  Broader  TTMs  capture 
efficiencies  from  training  these  tasks  together,  whereas  more  specific 
TTMs  may  yield  clearer  pictures  of  personnel  utilization  patterns. 
Clearly,  implementation  of  the  system  using  a  good  approximation  of  TTMs 
will  aid  in  identifying  the  characteristics  that  TTMs  should  possess  to  be 
maximally  useful  for  Air  Force  training  decision-aiding. 

The  validation  work  conducted  in  423X1  and  305X4  indicates  that  the 
recommended  TTM  construction  method  produces  good  first  approximations  of 
TTMs.  The  comparisons  among  independent  interpretations  of  the  task 
co-performance  cluster  diagram,  the  comparisons  among  independent  SME 
refinements  of  the  initial  TTMs,  the  comparisons  among  the  initial  and 
refined  TTMs,  and  the  task  analysis  of  selected  TTMs  all  support  the 
acceptance  of  the  recommended  TTM  construction  procedure  as  a  reliable  and 
valid  technique  for  task  clustering.  Statistically,  all  of  the  com¬ 
parisons  made  have  been  highly  significant.  Practically,  the  procedure 
produces  results  that  experienced  personnel  in  the  various  career  fields 
believe  capture  training  efficiencies,  and  does  so  in  a  timely  and 
cost-effective  manner. 


3.0  TASK  TRAINING  MODULE  ALLOCATION 

This  section  describes  the  R  &  D  of  the  TTM  allocation  component,  the 
second  of  two  components  of  the  TCS.  The  purpose  of  the  TTM  allocation 
component  in  the  TDS  is  to  provide  descriptions  of  alternative  ways  of 
allocating  training  among  classroom  instruction,  correspondence  courses, 
field  training  groups,  and  OJT.  The  allocation  component  should  be 
capable  of  describing  the  current  allocation  of  training  on  a  TTM  among 
these  training  settings  and  the  "most  preferred"  allocation  of  training, 
as  well  as  a  broad  range  of  other  possible  alternatives.  The  chief 
restriction  on  any  TTM  allocation  is  that  the  training  provided  in  all  of 
the  settings  be  sufficient  to  permit  the  average  airman  to  perform  all  of 
the  tasks  in  the  TTM  to  an  acceptable  level  (able  to  perform  the  tasks 
with  a  minimum  amount  of  assistance  and  supervision).  For  the  purposes  of 
the  TDS,  this  just-able- to- perform  level  of  proficiency  was  defined  as  the 
minimum  training  standard  for  TTMs. 
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Two  conceptual  issues  arise  from  the  requirement  that  allocations 
faithfully  represent  programs  capable  of  providing  training  to  a  minimum 
proficiency  level.  First,  what  is  the  best  method  of  describing  training 
settings  to  facilitate  gathering  allocation  information  from  SMEs?  And 
second,  what  type  of  measure  should  be  used  to  describe  an  amount  of 
training  in  a  setting?  The  first  part  of  this  section  discusses  the 
pilot-testing  that  was  conducted  to  address  these  questions.  In  the 
second  part,  the  results  of  applying  the  selected  TTM  allocation 
methodology  are  described.  The  next  section  discusses  a  preference 
estimation  procedure  that  is  based  on  the  allocation  survey  results. 
Finally,  issues  of  resource  requirements  to  apply  the  selected  allocation 
methodology  are  considered. 

3.1  Development  of  TTM  Allocation  Procedures 

Perhaps  the  thorniest  issue  faced  in  the  development  of  TTM  allocation 
methods  was  the  selection  of  a  measure  to  describe  proficiency  gains  from 
training  in  each  setting.  The  TDS  Statement  of  Work  (SOW)  called  for 
"...module  estimates  as  to  the  level  of  proficiency  recommended  for  each 
TTM  or  task  within  each  setting...."  (p.  13).  In  addition  to  proficiency 
levels  for  each  TTM  by  setting,  training  times  by  setting  must  also  be 
obtained,  for  use  in  the  estimation  of  training  costs. 

Major  research  programs  are  underway  throughout  the  armed  forces  to 
develop  reliable  and  valid  performance  measurement  techniques.  Much  of 
this  work  relies  heavily  on  detailed  task  analyses  and  identification  of 
the  specific  skills  and  knowledges  needed  to  perform  particular  tasks. 
Such  analyses  would  provide  a  wealth  of  information  concerning  the 
proficiency  gains  that  might  be  expected  with  a  given  amount  of  training 
of  a  particular  type.  When  such  information  is  available,  the  TDS  may  be 
able  to  incorporate  it  into  the  TTM  allocation  component.  In  the  short 
term,  however,  the  work  documented  in  this  report  suggests  that  TTM-by- 
setting  proficiency  estimates  can  be  gathered  efficiently  by  survey  and 
that  these  estimates  are  sufficiently  consistent  to  be  of  considerable 
value  to  the  users  of  the  TDS. 

Three  measures  of  ITM-by-setting  proficiency  were  proposed  and 
pilot-tested.  The  first  was  a  six-point  proficiency  rating  scale, 
anchored  with  "the  trainee  can  perform  no  aspects  of  the  task 
independently,  requires  constant  direction"  to  "the  trainee  can  perform 
all  aspects  of  the  tasks  to  minimum  standards  without  direct  super¬ 
vision."  This  rating  scale  equated  partial  proficiency  with  partial 
mastery  of  various  aspects  of  the  tasks  in  a  TTM  and  the  level  of 
supervision  required.  SMEs  were  also  required  to  provide  estimates  of  the 
number  of  hours  of  training  needed  to  achieve  the  rated  level  of  proficiency 
in  each  training  setting. 

The  second  measure  of  proficiency  proposed  and  field-tested  was  a 
direct  estimate  of  the  percentage  of  full  proficiency  achieved  in  a 
setting.  Full  (100%)  proficiency  for  the  purposes  of  the  questionnaire 
was  defined  as  "able  to  perform  the  tasks  to  minimum  acceptable  standards 
without  direct  supervision."  SMEs  were  to  then  indicate  the  percentage 
(0  to  100)  that  is  currently  or  should  be  provided  in  a  training  setting 
and  the  number  of  hours  required  to  achieve  that  level  of  proficiency. 
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The  third  and  final  proficiency  measurement  approach  was  to  estimate 
proficiency  from  the  training  time  estimates  provided  by  SMEs.  Each 
allocation  of  training  time  to  the  settings  provides  an  estimate  of  a 
training  program  that  is  sufficient  to  train  the  average  airman  to  minimum 
standards  on  the  tasks  in  the  TTM.  A  series  of  these  allocations  provides 
a  set  of  simultaneous  equations  that  could,  theoretically,  be  solved  using 
simple  algebra.  Each  estimate,  however,  will  have  some  error  associated 
with  it,  and  so,  statistical  methods  of  estimating  those  equation 
parameters  were  proposed. 

The  second  conceptual  issue  in  the  development  of  a  TTM -to- setting 
allocation  methodology  was  how  best  to  describe  training  settings.  The 
current  training  settings  (e.g.,  FTDs,  CDCs,  basic  or  advanced  courses) 
could  be  used.  There  are,  however,  several  complications  to  using  these 
actual  training  settings.  First,  there  is  variation  among  AFSs  as  to  the 
training  settings  used.  Second,  having  SMEs  rate  proficiency  gains  or 
training  times  in  actual  training  settings  may  interfere  with  estimating 
the  same  numbers  for  alternative  allocations.  What  is  of  interest  in 
alternatives  to  the  current  allocation  are  proficiency  gains  or  training 
times  using,  for  example,  a  restructured  CDC  or  a  new  FTD. 

As  an  alternative  to  using  actual  training  settings,  a  categorization 
of  types  of  training  was  proposed  and  field-tested.  The  types  of  training 
were  as  follows: 

Classroom  instruction  involving  lecture/discussion  and  related  reading 
(most  resident  technical  training). 

Correspondence  courses,  self-paced,  individual  study  from  text  (all 
CDCs). 

Hands-on  experience  in  small,  supervised  training  groups  using 
simulators,  mockups,  or  actual  equipment  (most  FTDs). 

Hands-on  experience  on  the  job  including  observing  others,  practicing 
the  tasks,  and  receiving  direction  (qualification  and  upgrade 
training) . 

In  addition  to  these  three  methods  of  estimating  proficiency  and  two 
ways  of  describing  training  settings,  two  different  questionnaire  formats 
were  proposed.  One  format  put  all  of  the  different  types  of  estimates  for 
a  TTM  on  a  single  page.  In  the  field-tested  questionnaires,  this  included 
four  different  estimates:  (a)  the  current  allocation,  (b)  the  ideal 
allocation,  (c)  the  minimum  entering  background,  and  (d)  the  maximum 
training  level.  The  current  allocation  was  to  describe  the  present 
training  program,  while  the  ideal  indicated  how  the  SME  believed  training 
should  be  allocated.  The  minimum  entering  background  was  the  level  of 
knowledge  (if  any)  an  airman  should  have  before  beginning  training  in  each 
setting.  Finally,  the  maximum  training  level  was  the  proficiency  that 
could  be  effectively  reached  in  a  setting.  The  alternative  format  was  to 
separate  the  different  proficiency  estimates  into  different  sections  of 
the  questionnaire. 

The  first  field  test  of  the  TTM  allocation  procedures  was  conducted  in 
AFS  811XX  at  Lackland  AFB,  TX.  The  two  allocation  booklets  completed  by 
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the  AFS  811XX  SMEs  both  used  the  actual  training  settings  and  the  format 
in  which  all  ratings  appeared  on  a  single  page.  They  differed  by  the 
method  of  measuring  proficiency:  One  used  a  proficiency  rating  scale;  the 
other  used  a  percent  of  full  proficiency  achieved.  Examples  of  the  survey 
formats  are  shown  in  Appendix  B.  The  results  of  the  field  test  can  be 
summarized  as  follows: 

1.  The  SMEs  were  not  comfortable  using  either  measure  of  proficiency. 

2.  Training  time  estimates,  which  were  given  in  conjunction  with  the 

proficiency  estimates,  were  considered  difficult  judgments,  but 

were  meaningful. 

3.  The  questionnare  format,  which  required  multiple  different  types  of 

judgments  on  a  single  page,  was  found  to  be  confusing. 

4.  Estimates  of  minimum  entering  background  were  largely  invariant  and 

generally  not  important  in  making  allocation  decisions. 

5.  The  use  of  actual  training  settings  interfered  with  proficiency 

estimation. 


In  particular,  the  SMEs  cited  difficulties  estimating  an  ideal 
allocation  using  existing  training  formats.  For  example,  how  could  the 
existing  career  development  course  (CDC)  yield  more  (or  less)  proficiency 
than  was  achieved  under  the  current  system? 

With  this  feedback,  the  TTM  Allocation  Questionnaire  was  revised.  In 
the  third  version,  SMEs  were  to  provide  only  training  time  estimates,  so 
proficiency  could  be  statistically  estimated.  The  minimum  entering 
background  estimates  were  omitted  and  different  types  of  judgment-^ 
(current  allocation,  ideal  allocation,  and  maximum  training)  were 
separated  into  different  sections  of  the  questionnaire.  Finally,  types  of 
training,  rather  than  the  actual  training  settings,  were  used;  the 
instructions  were  revised;  and  examples  of  ratings  were  included. 

This  questionnaire  was  taken  to  Keesler  AFB,  MS,  and  was  field-tested 
in  AFS  328X4.  Overall,  the  questionnaires  were  well  received.  Training 
time  estimates  were  familiar  measures  for  the  SMEs  to  work  with,  and 
although  the  task  was  considered  to  be  difficult,  the  SMEs  indicated  it 
was  manageable. 

3.2  Deriving  TTM  Allocation  Functions 

To  make  the  task  of  estimating  TTM-by-setting  allocations  more 
manageable  for  the  SMEs,  the  TTMs  for  each  specialty  were  grouped  into 
allocation  booklets,  based  on  similar  functions.  The  groupings  were 
provided  by  SMEs;  where  the  groupings  were  too  large  for  a  single  booklet, 
the  group  was  subdivided  based  on  OS  duty  categories,  or  arbitrarily,  to 
form  two  booklets  covering  the  same  general  topic. 

With  the  exception  of  one  face-to-face  administration  in  AFS  811XX 
which  was  used  to  refine  the  administrative  details  of  allocation 
surveying,  all  of  the  questionnaires  were  mailed  to  the  SMEs.  Addressees 
were  identified  based  on  their  responses  to  the  most  recent  OS,  which 
indicated  they  had  performed  particular  TTMs.  Only  E-4s  and  above  were 
sent  surveys.  The  surveys  were  grouped  by  base  and  sent  to  the  base's 
Survey  Control  Officer,  with  the  instructions  that  if  an  SME  had  departed, ' 
the  survey  was  to  go  to  the  SME’s  replacement  (by  position  title). 


The  reliability  of  the  allocation  judgments  was  estimated  from  the 
proportional  amount  of  the  total  rating  variance  accounted  for  by  TTMs 
within  each  training  setting.  Omega  squared  (Keppel,  1982)  provides  an 
index  of  this  proportion  of  variation  explained.  Omega  squared  will  be  0 
when  none  of  the  rating  variance  is  explained  by  TTMs,  or  equivalently 
when  there  is  no  variation  in  average  training  time  from  TTM  to  TTM.  It 
will  be  1  when  all  of  the  rating  variance  is  explained  by  TTMs,  or 
equivalently  when  each  individual  judgment  for  a  TTM  is  the  same  as  every 
ether  judgment.  The  size  of  the  error  component  effectively  limits  the 
size  of  Omega  squared,  since  Omega  squared  can  be  thought  of  as  the  ratio 
of  TTM  variance  to  the  sum  of  TTM  plus  error  variance.  Consequently, 
Cohen  (1977)  has  argued  that  an  Omega  square  of  0.15  or  more  is  a  "large" 
effect  in  social  science  and  behavioral  research,  whereas  an  Omega  squared 
of  0.06  is  a  "medium"  effect.  Although  these  cutoffs  are  admittedly 
arbitrary,  they  do  give  some  perspective  to  the  interpretation  of  Omega 
squared.  Additionally,  it  should  be  noted  that  R(ll)  in  the  CODAP  program 
REXALL  (CORREL  in  ASCII  CODAP)  is  Omega  squared. 

Across  the  four  specialties  studied,  some  trends  in  the  patterns  in 
the  reliabilities  of  allocation  judgments  can  be  discerned.  First,  it 
might  be  expected  that  the  reliabilities  for  the  current  allocation  would 
be  higher  than  those  for  the  most  preferred  allocation  or  the  maximum 
training  time.  After  all,  the  latter  two  judgments  should  be  subject  to 
individual  preference,  while  the  current  allocation  is  a  matter  of 
objective  reality.  This  expectation  was  not  supported,  however.  Overall, 
reliabilities  of  the  most  preferred  allocation  and  maximum  training  times 
were  about  the  same  (both  with  average  Omega  Squared  values  of  C.109)  as 
that  for  the  current  allocation  (average  Omega  squared  of  0.101). 

Across  the  different  allocations,  judgments  of  the  time  spent  in 
correspondence/ self- study  courses  showed  the  lowest  reliabilities,  with  an 
average  Omega  squared  of  0.087.  Work  experience  time  ratings  were 
somewhat  more  consistent  than  self-study  ratings,  with  an  average  Omega 
squared  of  0.091.  Classroom  and  field  training  group  times  showed  the 
highest  reliabilities  overall.  The  average  Omega  squared  for  classroom 
ratings  was  0.126,  while  that  for  field  training  group  times  was  0.121. 

When  an  SME  specifies  a  set  of  training  times  that  he  or  she  believes 
are  sufficient  to  train  the  average  airman  to  minimum  standards  for  a  TTM, 
a  function  relating  training  hours  to  proficiency  is  defined.  For 
example,  if  an  SME  indicates  that  10  hours  in  the  classroom  and  12  hours 
on  the  job  are  necessary  to  reach  minimum  standards,  these  data  can  be 
represented  as: 


10  class-hours  +  12  work-hours  +  0  self-study-hours  + 

0  field-training-hours  =  100%  of  minimum  standards. 


Each  allocation  questionnaire  respondent  provided  two  allocations  of 
this  type,  one  for  current  training  and  one  for  the  most  preferred 
allocation.  In  addition,  each  respondent  indicated  how  long  it  would  take 
to  train  a  TTM  in  each  of  the  settings  (e.g.,  in  100  hours  on  the  job)  so 
that  the  average  airman  would  have  reached  minimum  standards.  These  data 
can  be  represented  by  the  following  equation: 
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100  work-hours  +  0  class-hours  +  0  self-study-hours  + 

0  field-training-hours  =  100%  of  minimum  standards. 


Of  course,  complete  training  cannot  always  be  provided  in  each 
individual  training  setting.  For  example,  many  TTMs  cannot  be  completely 
trained  in  the  classroom  or  using  correspondence  courses,  in  the  cases  in 
which  complete  training  cannot  be  provided  in  a  setting,  the  respondent 
was  to  indicate  the  maximum  percentage  of  full  proficiency  that  could  be 
achieved  in  the  setting  and  the  time  taken  to  reach  that  level.  Thus, 
each  questionnaire  respondent  provided  six  allocation  judgments  for  each 
TTM:  four  that  related  the  maximum  training  time  in  a  setting  to  the 
proficiency  level  reached,  one  for  the  current  allocation,  and  one  for  the 
most  preferred  allocation. 

It  was  hypothesized  that  proficiency  gain  from  training  in  a  setting 
would  be  greatest  initially  and  would  decline  as  more  training  was 
provided  in  that  setting.  Eventually,  there  would  be  no  more  gain  from 
providing  training  in  that  setting.  Thus,  the  predicted  relationship 
between  proficiency  and  time  in  a  training  setting  is  that  of  initial  gain 
followed  by  proficiency  leveling-off,  a  negatively  accelerated  curve. 
This  general  relationship  is  depicted  in  Figure  2.  The  curves  in  Figure  2 
are  not  those  for  any  particular  TTM;  rather,  they  only  illustrate  the 
expected  form  of  the  relationship.  This  relationship  can  be  modeled  by 
the  following  polynomial  regression  equation: 

Proficiency'  =  a  *  class-hours  -  b  *  class-hours**2  + 
c  *  self-study-hours  -  d  *  self-study-hours**2  + 
e  *  field-training-hours  -  f  *  f ield-training-hours**2  + 
g  *  work-hours  -  h  *  work-hours* *2 , 


where  "a"  through  "h"  are  coefficients  to  be  estimated  by  multiple  regres¬ 
sion,  **2  indicates  squaring,  and  the  regression  equation  is  constrained 
to  pass  through  the  origin  (there  is  no  constant  for  the  Y  intercept). 

This  model  involves  specific  hypotheses  about  the  nature  of  the 
relationship  between  setting  training  hours  and  proficiency. 
Specifically,  controlling  for  training  in  each  of  the  other  training 
settings,  the  first-order  parameter  is  specified  to  be  positive  and  the 
second-order  parameter  is  negative,  yielding  the  predicted  negatively 
accelerated  cu'  e.  Across  the  four  AFSs  studied,  this  statistical  model 
was  strongly  supported.  Statistical  estimates  consistent  with  the 
polynomial  regression  equation  were  found  in  well  over  90%  of  the  TTM 
allocation  curves  in  all  four  specialties.  There  are  two  additional 
sources  of  support  for  the  conceptualization  of  proficiency  gain  as  a 
negatively  accelerated  function  of  training  time  in  a  setting.  First,  the 
overall  fit  of  the  polynomial  regression  model  was  found  to  be  quite  good, 
averaging  over  65%  (multiple  R  squared)  in  AFSs  423X1  and  305X4.  Second, 
the  additional  variance  explained  by  the  second-order  terms  in  the 
allocation  equations  was  substantial  (approximately  15%  increase  in  R 
squared  for  these  specialties),  indicating  that  simple  linear  functions 
are  not  sufficient  to  describe  proficiency  gains  from  training  in  a 
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Figure  2.  Hypothesized  Relationship  Between  Hours  of  Training  in  a 
Setting  and  Proficiency  Gain. 


setting;  rather,  a  curvilinear  model  more  adequately  describes  this 
relationship. 

A  second  rather  consistent  finding  in  the  derivation  of  the  allocation 
functions  was  that  the  most  likely  deviation  from  the  polynomial  regres¬ 
sion  model  for  all  four  AFSs  involved  training  time  judgments  for  self- 
study.  Although  the  reason  for  this  deviation  from  the  hypothesized  model 
for  self-study  times  is  not  clear,  two  conjectures  have  been  advanced. 
First,  it  is  possible  that  the  data  are  not  sufficiently  detailed  or  there 
is  not  enough  information  to  detect  the  negatively  accelerated  curve. 
Given  the  generally  lower  reliability  of  self-study  time  judgments,  more 
data  would  be  required  to  detect  a  pattern.  Second,  it  is  also  possible 
that  the  relationship  between  proficiency  and  self-study  time  is 
fundamentally  different,  either  in  nature  or  degree,  than  for  the  other 
training  settings.  For  instance,  since  the  lengths  of  training  are 
relatively  short  and  the  proficiency  gains  rather  small  for  self-study,  it 
may  be  that  a  simple  linear  equation  sufficiently  describes  this 
relationship  for  the  range  of  training  times  considered. 


3.3  Allocation  Preferences 

One  model  of  allocation  preference  estimation  might  be  called  the 
"rational"  model.  Specifically,  it  holds  that  preference  decreases 
linearly  as  the  distance  from  the  point  of  greatest  preference  increases. 
In  the  case  of  allocations  to  training  settings,  this  model  would  specify 
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that  preference  for  an  allocation  to  a  setting  should  decrease  as  the  time 
in  that  setting  varies  from  the  most  preferred  allocation.  It  would  also 
predict  that  preference  should  be  minimized  at  the  time  for  that  setting 
which  still  permits  complete  training  to  minimum  standards  but  which  is 
the  most  distant  point.  In  other  words,  preference  would  be  lcwest  at  the 
least  effective,  though  still  adequate,  allocation  of  training  to  a 
setting. 

Before  briefly  describing  how  this  model  was  implemented  for  TTM 
allocation  preference  estimation,  the  rational  model  should  be  compared  to 
an  alternative  procedure — the  empirical  approach.  Perhaps  the  most 
straightforward  preference  estimation  procedure  would  be  to  collect 
preference  information  on  a  number  of  different  allocations,  and  then 
empirically  derive  functions  relating  training  time  in  a  setting  to 
preference.  A  requirement  for  the  empirical  approach  is  that  a  means  of 
generating  alternative  allocation  examples  exist;  thus,  preference 
surveying  would  have  to  follow  allocation  surveying.  Cue  to  time 
limitations,  this  approach  was  not  used.  If  the  rational  model  is  to  be 
considered  a  viable  approximation  of  preferences,  however,  it  must  be 
validated  with  actual  preference  ratings. 

The  preference  measure  developed  for  allocations  of  TIM  training  to 
settings  is  a  four-point  scale,  and  this  index  will  be  4.0  only  for  the 
most  preferred  allocation.  For  any  deviation  from  the  most  preferred 
allocation,  the  preference  index  will  decrease,  and  the  amount  it 
decreases  depends  on  the  relative  distance  each  setting's  training  time  is 
from  the  most  preferred  level. 

An  example  will  illustrate  this  measurement  scheme.  Suppose,  for 
simplicity,  that  the  most  preferred  allocation  for  a  TIM  involved  only  two 
settings,  the  classroom  and  work  experience,  and  we  wish  to  estimate  the 
preference  for  an  alternative  allocation  also  involving  only  the  classroom 
and  work  experience  settings.  To  estimate  the  relative  change  from  the 
most  preferred  training  level  in  a  setting,  the  largest  possible  change 
from  the  most  preferred  level  can  be  estimated  from  that  level  and  either 
(a)  the  maximum  effective  time  (training  beyond  this  level  in  this  setting 
has  no  appreciable  effect)  or  (b)  the  minimum  required  time  for  a  setting 
(at  least  this  much  training  in  this  setting  is  required  to  reach  minimum 
standards) .  Ihe  largest  possible  change  is  the  larger  of  the  differences 
between  the  most  preferred  allocation  and  the  maximum  effective  time  or 
between  the  most  preferred  allocation  and  the  minimum  required  time.  For 
this  hypothetical  example,  the  hours  of  training  in  the  most  preferred 
allocation,  in  an  alternative  allocation,  the  maximum  effective  time,  and 
the  minimum  required  time  are  as  follcws: 

Classroom  Self-Studv  Field  Tmg  Work  Exp 


Most  preferred  allocation 

20 

0 

0 

45 

Alternative  allocation 

30 

0 

0 

40 

Maximum  effective  time 

30 

10 

15 

100 

Minimum  required  time 

0 

0 

0 

20 

21 


With  these  data,  the  largest  possible  deviation  from  the  most 
preferred  level  of  training  in  the  classroom  is  20  hours  (the  difference 
between  the  most  preferred  allocation  and  the  minimum  required  time).  The 
actual  change  for  the  alternative  allocation  was  10  hours,  or  a  relative 
change  of  0.50  ((20  -  10)  /  20),  one-half  of  the  largest  possible  change. 
The  largest  possible  change  from  the  most  preferred  level  for  work 
experience  is  55  hours  (the  difference  between  the  most  preferred  and  the 
maximum  effective  time).  The  actual  change  in  the  alternative  was  5 
hours,  for  a  relative  change  of  about  0.91  ((55  -  5)  /  55).  By  summing 
the  relative  changes  from  the  most  preferred  level  of  training  tor  each 
training  setting,  a  preference  estimate  of  3.41  is  obtained:  1.0  for 
self-study  (which  remained  unchanged),  plus  1.0  for  field  training  (which 
remained  unchanged),  plus  0.50  for  classroom  training,  plus  0.91  for  work 
experience. 

All  of  the  data  points  used  in  the  calculation  of  this  preference 
index  were  collected,  or  are  derivable  from,  data  collected  with  the 
allocation  survey.  The  most  preferred  allocation  and  maximum  training 
times  were  estimated  by  the  survey  respondents.  It  will  be  necessary, 
however,  to  "standardize"  these  times  for  the  most  preferred  allocation, 
so  that  the  times  yield  exactly  100%  proficiency.  Of  course,  any  rating 
will  contain  some  error,  so  that  the  rating  of  the  most  preferred 
allocation  may  yield  slightly  more  or  slightly  less  than  full 
proficiency.  Either  case  makes  preference  estimation  problematic,  since 
the  index  measures  relative  movement  along  the  allocation  curve.  To 
resolve  this  problem,  individual  setting  times  can  be  converted  to 
relative  proportions  of  the  total  time.  Then  the  total  time  can  be 
adjusted,  so  that  the  sum  of  the  proficiency  gains  from  each  setting  is 
exactly  100%. 

Finally,  with  estimates  of  the  most  preferred  allocation  that  specify 
exactly  100%  proficiency  and  maximum  training  times,  the  only  other 
datapoint  required  for  preference  estimation  is  the  minimum  training  time 
required  in  each  setting.  This  time  is  easily  derived  from  the  maximum 
proficiencies  for  all  other  settings.  If  the  maximum  proficiencies  from 
all  other  settings  are  summed,  the  minimum  proficiency  required  from  a 
setting  is  the  difference  between  this  sum  and  100%  (if  the  sum  is  less 
than  100%).  If  the  sum  is  100%  or  greater,  there  is  no  minimum 
proficiency  requirement  for  that  setting,  or  in  other  words,  training  to 
minimum  standards  can  be  achieved  without  use  of  that  setting. 

In  sum,  the  procedures  developed  to  obtain  allocation  judgments  can 
be  used  effectively  to  estimate  current  and  ideal  training  patterns  and  to 
describe  relative  deviation  from  the  most  preferred  allocation.  An 
additional  important  criterion  for  evaluating  the  practicality  of  these 
methods  is  the  resource  requirements  for  applying  them.  This  issue  is 
addressed  in  the  section  that  follows. 


3.4  Resource  Requirements 

The  estimates  of  resource  requirements  for  deriving  TTM  allocations 
are  based  on  material  costs  and  personnel  salaries  from  Federal  Service 
and  Air  Force  salary  schedules.  They  do  not  include  charges  for  computer 
time,  salaries  for  support  personnel  such  as  secretarial  staff,  or  any 


travel  expenses  which  might  be  required  to  convene  groups  of  SMEs  or  to 
have  TDS  personnel  meet  with  SMEs  at  remote  bases. 

In  general,  the  process  of  deriving  allocation  functions  for  a 
specialty  can  be  divided  into  three  steps:  (a)  TTM  allocation  booklet 
development;  (b)  administration  of  the  surveys;  and  (c)  data  analysis. 
Once  TTMs  were  available  for  a  specialty,  the  first  step  in  allocation 
booklet  development  was  to  have  SMEs  group  the  TTMs  according  to 
similarity.  These  groupings  of  TTMs  then  became  the  basis  for  forming 
separate  allocation  booklets,  which  could  be  distributed  to  SMEs  with 
experience  in  the  tasks  referenced  in  them.  Also  part  of  this  step  is 
assembling  the  booklets  and  duplicating  sufficient  copies  (20  per 
booklet).  The  total  cost  of  the  TTM  allocation  booklet  development  step 
was  estimated  to  be  $1,750.45  to  $2,080.67,  depending  on  the  diversity  in 
the  career  field. 

The  second  step  was  survey  administration.  All  of  the  allocation 
booklets  for  a  base  were  mailed  to  the  Survey  Control  Officer,  who  handled 
distribution,  collection,  and  return  of  the  completed  questionnaires. 
During  booklet  assembly  and  shipping,  and  in  the  subsequent  weeks  when 
completed  surveys  were  being  returned,  an  analyst  would  be  responsible  for 
overseeing  booklet  preparation  and  monitoring  return  rates.  This  respon¬ 
sibility  would  include,  as  required,  answering  questions  concerning  the 
survey  from  either  the  Survey  Control  Officer  or  the  respondents.  The 
cost  of  this  step  was  estimated  at  $876.63 

After  all  of  the  completed  surveys  were  received,  the  data  analysis 
step  was  begun.  This  step  included  data  entry/keypunching,  statistically 
analyzing  the  data,  and  interpreting  and  documenting  the  findings.  The 
basic  statistical  analyses  would  include:  (a)  data  verification,  including 
range  tests  and  internal  consistency  checks;  (b)  reliability  estimation, 
including  identification  and  removal  of  outliers;  (c)  generation  of 
descriptive  statistics  on  the  current  and  most  preferred  allocations;  and 
(d)  derivation  of  allocation  functions.  The  data  verification  step  was 
handled  by  software  designed  for  this  purpose,  but  could  also  have  been 
done  with  the  interactive  statistical  package  that  was  used  for  the 
reliability  estimation,  descriptive  statistics,  and  multiple  regression 
analysis.  The  total  cost  of  this  step  was  estimated  at  $2,373.73. 

Thus,  the  total  cost  for  applying  the  TTM  allocation  procedures 
discussed  above  ranges  from  $5,000.81  to  $5,331.03,  depending  on  the 
diversity  in  the  career  field.  Again,  these  estimates  do  not  include 
computer  time  charges,  salaries  for  support  personnel,  or  travel  expenses 
(if  any).  It  should  also  be  noted  that  they  are,  at  best,  a  rough 
approximation  of  what  it  would  cost  to  apply  the  allocation  data 
collection  procedures  to  a  new  AFS.  More  definitive  estimates  are 
difficult  to  provide  since  previous  work  has  blended  data  collection  and 
analysis  with  R  &  D. 


3.5  Conclusions 


The  R  &  D  on  the  four  specialties  studied  has  identified  a  survey 
procedure  that,  with  limited  numbers  of  respondents,  can  yield  adequately 
stable  estimates  of  proficiency  gains  from  hours  of  specified  types  of 
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training.  The  allocation  and  preference  functions  derived  from  this 
survey  procedure  give  the  TDS  maximum  flexibility  in  considering  different 
ways  of  dividing  training  among  training  settings,  as  well  as  identifying 
the  limits  of  each  type  of  training.  Difficult  issues  such  as  proficiency 
measurement  and  the  description  of  partial  allocations  of  training  to 
different  training  settings  were  generally  resolved  to  the  satisfaction  of 
the  many  SMEs  who  have  devoted  their  time  to  the  development  of  these 
procedures.  Nonetheless,  the  procedures  should  continue  to  be  developed. 
In  the  short  term,  work  could  be  directed  at  refining  the  allocation 
survey  procedures.  For  example,  AFS-specific  examples  of  each  type  of 
training  might  facilitate  accurate  ratings.  In  the  long  term,  alter¬ 
natives  or  supplements  to  this  basic  survey  methodology  should  be 
evaluated.  The  results  of  major  research  efforts  to  measure  work 
performance,  for  example,  might  be  integrated  into  the  methods  used  to 
estimate  training  proficiency  gains,  to  the  improvement  of  the  TDS.  It  is 
these  directions  for  future  research  into  enhanced  TCS  methodologies  to 
which  the  final  section  of  this  report  is  devoted. 


4.0  DIRECTIONS  FOR  FUTUPE  WORK 

This  section  is  divided  into  two  parts,  directions  for  future  work  in 
TTM  construction  and  in  TTM  allocation.  Additionally,  each  part  progres¬ 
ses  from  relatively  minor,  but  strongly  recommended  modifications  of  the 
basic  data  collection  or  analysis  procedures  to  basic,  long-term  research 
into  improved  TCS  methodologies.  The  former  changes  would  result  in 
immediate  improvement  in  the  quality  of  the  TCS  databases.  The  future 
research  directions,  on  the  other  hand,  have  the  potential  to 
significantly  or  totally  alter  TCS  methodologies. 


4.1  TTM  Construction 

4 .1.1  Sensitivity  Analysis  of  the  TDS  to  Changes  in  TTMs 

Certainly  one  of  the  most  important  criteria  by  which  the  adequacy  of 
the  TTM  construction  methods  can  be  evaluated  is  the  degree  to  which  TTMs 
fulfill  their  intended  roles  in  the  TDS.  Chief  among  these  roles  are  that 
they  be  appropriate  for  describing  patterns  of  personnel  utilization  and 
training  (both  resident  and  field  training,  both  current  and  alternative 
patterns),  and  that  they  encompass  economies  of  training  tasks  together. 
The  question  of  the  specificity  of  TTMs  can  be  addressed,  consequently,  by 
varying  the  breadth  of  the  TTMs  and  evaluating  the  impact  on  Field 
Utilization  Subsystem  (FUS)  and  Resource/ Cost  Subsystem  (RCS)  products. 
This  analysis  of  the  sensitivity  of  TDS  outputs  to  the  specificity  of  TTMs 
would  yield  a  great  deal  of  information  on  how  TTMs  should  be  formed  to 
meet  the  requirements  of  the  operational  system. 

4.1.2  Nonhierarchical  Clustering 

Nonhierarchical  clustering  promises  to  provide  some  of  the  flexibility 
that  hierarchical  clustering  lacks.  Specifically,  nonhierarchical  clus¬ 
tering  should  permit  tasks  not  classified  by  hierarchical  techniques  to  be 
grouped  with  the  most  similar  tasks.  It  also  allows  tasks  to  appear  in 
more  than  one  TTM.  Nonhierarchical  clustering  should  be  fully  evaluated 


as  a  TTM  construction/refinement  technique  as  soon  as  the  procedure  is 
fully  operational. 

4.1.3  Integration  with  Other  Task  Characteristics  Research 

Considerable  research  is  being  conducted  which  has  implications  for 
task  clustering  technology.  Much  of  this  work  has  been  identified  and 
described  elsewhere  (e.g.,  Mitchell,  Sturdevant  et  al.,  1987;  Vaughan  et 
al.,  1985),  and  the  TDS  is  being  designed  so  that  it  can  make  productive 
use  of  these  techniques.  However,  potentially  valuable  interfaces  between 
the  TDS  and  other  systems  will  not  occur  unless  research  is  conducted  to 
assess  the  interdependencies  and  to  integrate  the  technologies. 

4.1.4  Task  Characteristics  Surveys 

Task  clustering  based  on  task  characteristics  should  be  investigated. 
Research  on  clustering  tasks  using  OS  background  characteristics  was 
evaluated;  unfortunately,  these  data  are  not  gathered  for  this  purpose  and 
it  is  not  possible  to  directly  link  OS  background  characteristics  to 
tasks.  To  fully  test  the  adequacy  of  task  clustering  based  on  equipment 
needed  to  accomplish  the  tasks,  procedures/tactics  used,  or  theories/facts 
that  should  be  known,  data  which  specifically  link  tasks  with  these 
characteristics  should  be  gathered.  The  number  of  tasks  and  possible 
characteristics  is  large,  however,  making  even  simple  tabulation  of  such 
linkages  problematic.  On  the  other  hand,  it  might  be  possible  for  SMEs  to 
indicate  the  important  equipment,  procedures,  or  knowledges  for  subsets  of 
tasks.  Additionally,  information  from  senior  personnel  would  not 
necessarily  be  required  (except  for  tasks  they  perform),  and  so,  the 
applicable  pool  of  respondents  would  be  substantially  increased. 

4 . 1 . 5  Construction  of  Knowledge  TTMs 

Occupational  survey  tasks  are,  by  design,  statements  of  observable 
actions  incumbents  perform  on  the  job.  Such  behaviorally  oriented 
statements  are  optimal  for  describing  the  structure  of  work.  They  may  be 
less  appropriate,  however,  for  determining  training  requirements,  since 
knowledge  requirements  for  the  tasks  are  not  explicitly  delineated.  For 
example,  knowledge  of  certain  electronic  principles  may  be  required  for 
performing  a  variety  of  different  technical  maintenance  tasks,  and  OS  task 
groupings  may  cut  across  such  lines  of  distinction.  Another  example  is  in 
the  area  of  contingency  training,  where  occupational  survey  tasks  which 
reflect  day-to-day  operations  do  not  indicate  the  knowledges/skills 
necessary  for  emergency  situations.  Since  different  training  settings  may 
be  more  appropriate  for  imparting  the  knowledge  component  of  task 
performance  than  the  procedural  component,  allocation  decisions  for  these 
different  types  of  TTMs  may  vary. 

Constructing  TTMs  composed  of  relevant  knowledges  (e.g.,  electronic 
principles,  emergency  situations)  would  seem  to  be  an  appropriate  course 
of  action.  This  approach  would  substantially  increase  the  amount  of  data 
that  would  be  required  by  the  TDS,  as  catalogs  of  relevant  knowledges, 
where  they  were  not  available,  would  have  to  be  developed.  Given  such  a 
list,  a  means  of  clustering  would  have  to  be  developed;  the  procedures  > 
documented  in  Perrin  et  al.  (1986)  should  serve  as  a  useful  starting  ’ 
point  in  this  work.  The  clustering  of  knowledges  would  then  have  to  be 
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tied  to  the  task  modules;  i.e.,  the  knowledge  clusters  that  are  required 
for  the  performance  of  each  TTM  would  have  to  be  identified.  Separate 
cost  information  would  have  to  be  developed  for  each  of  the  knowledge 
groupings.  With  these  data,  personnel  utilization  patterns  could  describe 
required  training  for  job  performance  in  terms  of  both  the  knowledge  and 
procedural  requirements,  and  training  decisions  could  take  into  account 
economies  in  both  areas. 

4.1.6  Tasks  as  the  Units  of  Analysis 

The  key  to  making  a  task-based  approach  workable  is  to  develop  a 
complete  task  similarity  matrix  that  reflects  training  economy  and 
specifically,  economy  in  training  time.  Skills  and  knowledge  similarity 
would  appear  to  serve  such  a  function.  Where  there  is  considerable 
latitude  to  group  tasks  in  various  ways,  such  as  in  the  basic  resident 
course,  blocks  of  instruction  may  represent  relatively  homogeneous 
clusters,  and  training  time  may  be  minimized  as  instruction  on  one  task 
builds  on  what  has  already  been  learned  about  another.  When  practical  or 
situational  limits  exist,  such  as  the  constraints  job  requirements  place 
on  what  is  trained  in  the  work  environment,  the  tasks  that  must  be  trained 
together  may  be  a  more  heterogeneous  group,  and  so,  fewer  training  time 
economies  can  be  realized.  In  ether  words,  the  time  to  train  the  group  of 
tasks  in  the  first  case  may  be  substantially  less  than  the  time  to  train 
each  of  them  separately,  whereas  in  the  second  case,  the  total  time  to 
train  may  approach  the  sum  of  the  individual  times. 

The  advantage  of  this  approach  is  that  the  U  &  T  pattern,  current  or 
alternative,  would  specify  which  tasks  were  grouped  for  each  type  of 
training,  and  the  similarity  of  the  tasks  could  then  be  used  to  specify 
the  time  economies  of  training  the  tasks  together.  Additionally,  it  is 
not  clear  that  this  approach  would  significantly  change  or  increase  the 
amount  of  information  needed  by  the  TDS.  For  example,  tasks  which  have 
many  skills  and  knowledges  in  common  probably  have  similar  training  time 
requirements,  so  that  such  data  could  be  collected  for  representative, 
rather  than  all  tasks.  Given  the  flexibility  this  approach  would  afford 
the  TDS,  these  issues  deserve  closer  study  and  evaluation. 


4.2  TTM  Allocation 

4.2.1  TTM  Grouping  for  Allocation  Booklets 

The  method  used  to  group  TTMs  into  manageable  units  for  allocation 
surveying  was  to  have  SMEs  sort  them  according  to  similarity.  This 
procedure  is  relatively  expensive,  especially  if  travel  expenses  are 
required,  and  proved  to  be  ineffective  in  several  cases.  Perhaps  the  best 
example  of  the  difficulty  encountered  using  this  approach  was  in  AFS  423X1 
for  the  Oxygen  Systems  Maintenance  allocation  booklet.  TTM  31,  Maintain 
Chemical  Oxygen  Generator,  was  placed  in  this  booklet  along  with  four 
other  TTMs  (numbers  17,  22,  26,  and  28).  The  response  rate  on  the  latter 
fair  TTMs  ranged  from  7  to  9  raters;  no  nonzero  responses  were  obtained  on 
TTM  31  (only  two  respondents  gave  any  ratings  at  all).  Clearly,  personnel 
that  perform  the  other  four  TTMs  are  unlikely  to  perform  TTM  31.  This  was 
not  an  isolated  problem;  similar  problems  were  noted  in  the  other  AFSs. 
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The  problem  is  that  SMEs  do  not  necessarily  group  TTMs  according  to 
co-performance,  so  that  if  airmen  perform  one  of  the  TTMs  in  the  booklet, 
they  are  also  likely  to  be  familiar  with  the  others  in  that  book.  The 
result  is  that  certain  TTMs  in  the  allocation  booklets  have  very  high 
response  rates  while  ethers  have  low  rates.  Perhaps  this  problem  could  be 
reduced  by  directing  the  SMEs  to  group  the  TTMs  according  to  co¬ 
performance;  it  is  not  clear,  however,  that  this  approach  would  be 
entirely  successful.  The  research  reviewed  earlier  on  TTM  construction 
indicates  that  SMEs  groupings  can  be  approximated  by  co- performance,  and 
that  there  is  variation  among  groups  of  SMEs  as  to  how  they  cluster  tasks 
given  the  same  sorting  instructions. 

Three  alternatives  to  the  current  method  of  grouping  TTMs  into 
allocation  booklets  are  preposed.  The  first  is  to  provide  each  allocation 
survey  respondent  with  a  TTM  reference  volume  that  lists  all  of  the  TTMs 
and  the  tasks  in  them.  The  respondent  would  then  be  asked  to  identify  the 
8  to  10  TTMs  with  which  they  were  most  familiar,  and  complete  an 
allocation  questionnaire  for  each  of  them.  An  approach  similar  to  this  is 
being  used  for  the  Training  Time  Questionnaire,  a  survey  instrument  used 
for  data  collection  in  the  FUS,  which  is  accompanied  by  a  TTM  reference 
volume. 

The  second  alternative  is  to  group  the  TTMs  according  to 
co-performance,  using  ASCII  CODAP  routines  for  co-performance  clustering. 
If  the  task  data  were  collapsed  into  TTM -level  data,  for  example,  by 
averaging  over  the  tasks  in  the  TTM,  it  would  be  possible  to  use  CO  DAP 
programs  to  cluster  TTMs.  The  TTM  co-performance  clusters  identified  from 
the  cluster  diagram  would  then  form  the  allocation  booklets.  ASCII  CO  DAP 
programs,  discussed  by  Phalen,  Staley,  and  Mitchell  (1987),  offer  another 
option  to  clustering  TTMs.  They  demonstrate  a  method  of  displaying  a 
person  by  task  matrix,  with  both  the  individuals  and  the  tasks  in  TPATH 
(clustered)  order.  This  product  permits  one  to  investigate  the 
relationship  between  jobs  (person  clusters)  and  TTMs  (task  clusters). 
Such  a  product  should  be  useful  for  refining  task  clusters  to  form  TTMs 
and  identifying  the  personnel  performing  the  TTMs,  so  that  allocation 
booklets  can  be  mailed  to  appropriate  individuals. 

The  final  alternative  is  to  customize  or  individualize  the  allocation 
booklet  sent  to  each  SME,  based  on  the  tasks  the  SME  reported  performing 
on  the  OS.  For  this  approach  to  work,  formation  of  TTMs  and  allocation 
surveying  should  follow  the  SME's  completion  and  return  of  the  OS  as 
closely  as  possible.  The  allocation  survey  sent  to  each  SME  would  contain 
only  those  .TTMs  the  SME  had  indicated  he  or  she  had  performed.  This 
approach  is  perhaps  optimal  in  targeting  the  most  qualified  airmen  for 
allocation  surveying.  It  does  increase  the  expense  incurred  in  preparing 
the  surveys,  since  it  is  possible  that  no  two  surveys  would  be  exactly  the 
same;  automating  survey  preparation  could  substantially  reduce  this 
expense,  however. 

4.2.2  Selecting  the  Survey  Sample 

In  conjuction  with  modifying  the  method  of  grouping  TTMs  into 
allocation  booklets,  the  method  of  selecting  the  SMEs  to  receive 
allocation  booklets  also  needs  to  be  refined.  The  method  that  has  been 
used  is  to  select  those  SMEs  who  reported  performing  the  highest 


proportion  of  the  tasks  in  a  booklet.  Unfortunately,  this  procedure  omits 
the  instructors  who  train,  but  do  not  actually  perform  the  tasks.  The 
result  was  that  for  all  of  the  AFSs  studied  except  811XX,  very  few 
trainers  were  included  in  the  survey  (in  AFS  811XX,  trainers  were  included 
in  a  group  administration  of  the  survey). 

Perhaps  the  simplest  approach  to  this  problem  would  be  to  compile  a 
TTM  reference  volume,  as  discussed  above,  which  would  be  sent  to  the 
appropriate  technical  training  centers.  Trainers  would  then  select  the 
TTMs  for  which  they  had  responsibility,  and  complete  allocation  surveys 
for  these  TTMs.  Again,  this  approach  is  being  used  for  the  Training  Time 
Questionnaire. 

4.2.3  Refine  the  Allocation  Survey  Instructions 

Although  it  is  almost  always  possible  to  improve  the  instructions  to  a 
relatively  complex  survey  instrument,  some  problems  have  occurred  with 
enough  regularity  to  warrant  changes  in  the  allocation  survey.  First, 
rated  training  times  of  zero  for  all  training  settings  has  occurred  rather 
frequently  in  all  four  AFSs  studied.  According  to  the  survey's  instruc¬ 
tions,  this  rating  should  indicate  that  no  training  is  required  to  reach 
minimum  proficiency  for  this  TTM.  In  practice,  it  may  mean  either  that 
the  SME  was  unfamiliar  with  the  TTM  or  that  the  SME  believed  no  one  in  the 
AFS  currently  performed  the  TTM.  Improved  sampling  procedures,  especially 
individualized  allocation  booklets,  should  make  it  less  likely  that  an  SME 
would,  due  to  lack  of  familiarity  with  the  TTM,  indicate  no  training  was 
required.  An  additional  precaution  against  this  confusion  might  be  to  add 
a  warning  concerning  this  situation,  such  as  the  following: 

Do  NOT  indicate  no  (zero)  training  times  for  all  types  of  training, 
INLESS  NO  TRAINING  IS  REQUIRED.  If  you  have  not  been  trained  on  the 
task  module,  please  write  DK  (Don't  Know)  on  the  page,  turn  to  the 
next  page,  and  continue. 

A  second  prevalent  problem  is  the  wide  variation  in  training  times 
reported,  especially  for  work  experience  times.  For  example,  one  SME 
provided  work  experience  time  estimates  in  access  of  20,000  hours  to  reach 
minimum  proficiency  on  certain  TTMs.  Although  it  was  possible  to  identify 
and  remove  the  highly  deviant  estimates,  it  would  be  desirable  to  reduce 
their  frequency  by  improving  the  instructions.  Discussions  with  SMEs 
following  group  administrations  of  allocation  surveys,  as  well  as  written 
feedback  on  the  mail  surveys,  have  indicated  that  the  difficulty  is  one  of 
calendar  time  compared  to  time-on-task  estimates.  Some  SMEs  appear  to 
have  provided  total  calendar  time  estimates  rather  than  time-on-task 
estimates.  The  former  is  the  total  elapsed  time  before  a  TTM  is  mastered; 
the  latter  is  the  time  spent  actually  working  on  or  studying  about  the 
TTM.  For  instance,  if  a  task  in  a  TTM  is  encountered  only  about  once  a 
year  and  must  be  performed  3  times  before  minimum  proficiency  is  reached, 
the  calendar  time  estimate  is  3  years.  If,  for  the  same  task,  8  hours  is 
needed  to  complete  the  task  the  first  time,  6  the  second,  and  5  the  third, 
the  total  time-on-task  is  19  hours.  A  precaution  against  this  problem 
might  take  the  form  of: 

Training  times  should  indicate  only  the  time  spent  learning  about, 
studying  about,  or  performing  the  tasks,  NOT  total  calendar  time. 


A  final  problem  noted  in  the  allocation  survey  procedures  was  some 
confusion  concerning  the  different  types  of  training.  Referents  for  some 
of  the  types  of  training  are  not  always  clear.  This  problem  was  perhaps 
most  apparent  in  the  AFS  811XX  survey  results.  The  clearest  referent  to 
field  training  groups  is  the  FTD.  In  AFS  811XX,  this  type  of  training  is 
not  provided;  consequently,  SMEs  in  811XX  tended  either  to  give  nonzero 
field  training  times  for  all  of  the  TTMs  they  rated  or  to  none  of  them 
(presumably  depending  on  whether  they  believed  any  of  their  training  fell 
under  the  general  title  of  field  training). 

.  The  solution  to  this  final  problem  is  not  as  straightforward  as  the 

I  ones  considered  previously.  One  possible  approach  would  be  to  list 

[  examples  of  specific  training  that  were  primarily  of  one  type  or  another. 

Or,  to  be  more  complete,  all  training  could  be  classified  as  primarily  of 
the  classroom,  self-study,  or  field  training  types.  Unfortunately,  much 
training  is  a  combination  of  two  or  three  of  these  types,  making 
»  classification  of  training  difficult.  It  might  be  possible  to  indicate 

the  parts  of  a  course  of  study  that  were  of  each  type;  e.g.,  a  particular 
course  might  consist  of  1  week  of  lecture  (classroom-type  training)  and  2 
weeks  of  hands-on  practice  (field  training).  In  addition  to  the 
difficulties  of  providing  a  meaningful  classification  of  training  into  the 
three  types  of  training,  the  classification  might  interfere  with  estimates 
of  the  most  preferred  allocation  or  maximum  training  times.  Such 
interference  was  noted  when  actual  training  settings  were  used  for 
allocation  judgments  rather  than  types  of  training.  Nonetheless,  this 
.  type  of  classification  scheme  holds  great  promise  for  improving  the 

consistency  and  accuracy  of  allocation  judgments  and  warrants  further 
consideration. 

4.2.4  Identifying  and  Handling  Outliers 

The  work  that  was  performed  on  identifying  outliers,  especially  in  AFS 
811XX,  indicated  that  an  SME's  judgments  often  were  inconsistent  with  the 
group's  ratings  for  only  one  or  two  of  the  types  of  training,  rather  than 
I  being  inconsistent  for  all  types.  Part-whole  correlations  (the  cor¬ 

relations  between  the  SME's  training  time  estimates  for  a  particular  TTM 
in  a  particular  setting  and  the  group  average  for  the  same  rating)  were 
often  positive  for  some  settings  and  negative  for  others.  In  other  words, 
the  SME  would  be  consistent  with  the  group's  estimate  for  certain  types  of 
training  (presumably  the  ones  with  which  he  or  she  was  familiar),  and 
inconsistent  with  others.  How  to  handle  this  differential  reliability  is 
a  question  for  further  research. 

One  approach  would  be  to  exclude  only  those  ratings  which  were  highly 
deviant  from  the  group  average.  For  example,  an  SME's  classroom  time 
estimates  which  were  highly  inconsistent  with  the  other  SMEs'  ratings  for 
the  same  TTMs  might  be  excluded  from  the  sample.  This  SME's  times  for 
self-study,  field  training,  and  work  experience  would  be  retained. 
Unfortunately,  for  the  derivation  of  the  allocation  curves,  this  approach 
has  the  same  effect  as  removing  all  of  the  SME's  ratings  (since  the  case 
is  dropped  from  multiple  regression  if  any  of  the  data  are  missing).  An 
alternative  is  to  replace  the  inconsistent  data  with  the  group  average  for 
that  TTM  and  setting.  Another  approach  would  be  to  use  a  weignted  average 
of  the  other  settings'  training  times  for  that  SME  or  of  task  character¬ 
istics  for  the  tasks  in  the  TTM  (the  weighted  average  being  based  on 
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multiple  regression).  Finally,  it  is  possible  to  develop  regression 
equations  from  a  correlation  matrix  that  is  based  on  partial  data.  Each 
of  these  approaches  has  its  limitations  and  each  has  been  studied  as  a 
general  method  for  handling  missing  data  in  multiple  regression.  In  the 
context  of  developing  allocation  curves,  these  methods  of  handling 
inconsistent  data  should  be  evaluated  to  identify  a  practiced  method  of 
correcting  for  differential  reliability. 

4.2.5  Empirical  Preference  Curves 

Functions  which  relate  setting  training  times  to  managerial 
preferences  should  be  developed  and  validated.  Using  the  allocation 
functions  that  have  been  developed,  a  number  of  alternative,  feasible 
allocations  of  training  could  be  generated  for  each  TTM.  Then,  preference 
ratings  for  each  alternative  could  be  obtained  from  appropriate  Air  Force 
training  managers  and  the  data  statistically  analyzed.  Using  this 
approach,  the  preference  tradeoffs  of  allocating  TTMs  in  different  ways 
could  be  specifically  modeled  and  represented  in  the  TDS,  much  the  same 
way  that  training  time  tradeoffs  are  currently  represented. 

4.2.6  Additional  Validation  Research 

Additional  work  on  validating  the  allocation  curves  should  be 
performed.  This  research  would  necessarily  focus  on  a  limited  number  of 
TTMs,  collecting  detailed  training  information  on  them.  These  training 
data  could  then  be  compared  to  the  survey  results.  Possible  sources  of 
bias  in  the  survey  results  could  be  identified  and  perhaps  remedied 
through  this  procedure. 

4.2.7  Further  Research  on  Time  to  Train 

Section  2.0,  Task  Training  Module  Allocation,  began  with  a  discussion 
of  some  of  the  conceptual  difficulties  involved  in  measuring  proficiency 
gain  from  training.  Chief  among  them  was  developing  a  method  of  measuring 
proficiency,  the  subject  of  extensive  research  throughout  the  armed 
services.  With  a  standardized,  objective  measure  of  performance  on  a 
task,  proficiency  gain  could  be  measured  as  the  change  from  before  to 
after  training  of  a  particular  type.  The  potential  payoff  for  the  TDS  of 
performance  assessment  research  in  the  Air  Force  is  substantial,  and 
efforts  should  be  made  to  ensure  that  the  results  of  performance 
assessment  research  are  incorporated  into  the  TDS,  as  appropriate. 

Research  on  integrating  and  supplementing  a  systematic  performance 
assessment  program  should  be  undertaken  in  the  near  future.  Most  of  the 
work  on  performance  measurement  is  at  the  task  level  or  lower  (the  skills 
and  knowledges  underlying  task  performance),  while  the  TDS  operates  at  a 
task  group  level  (TTMs).  Additionally,  performance  measurement  programs 
are  not  generally  concerned  with  time  to  train,  a  key  statistic  for  the 
TDS.  Methods  of  estimating  training  times  at  the  skill/Knowledge  level 
and  then  aggregating  those  estimates  to  the  TTM  level  should  be  developed 
and  evaluated.  Although  the  potential  benefits  to  the  TDS  of  a  systematic 
performance  program  are  substantial,  this  payoff  will  not  be  realized 
without  further  R  &  D  of  performance  assessment  technologies. 
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Quite  apart  from  the  performance  assessment  research,  further  re¬ 
search  on  time  to  train  TTMs  in  different  training  settings  could  1 2  of 
considerable  value  to  the  TDS.  For  example,  group  consensus  judgments 
could  be  compared  to  the  survey  averages  which  are  currently  being  used. 
Presumably,  group  judgments  would  be  less  prone  to  the  type  of 
differential  reliability  observed  in  the  survey  results,  since  a  group  is 
more  likely  to  have  art  least  one  person  familiar  witn  each  of  the  training 
settings. 


4.3  Priorities  of  Future  Research 

Some  hard  choices  must  be  made  among  these  various  future  R  &  D 
alternatives  since  it  is  unlikely  that  all  the  alternatives  outlined  could 
be  fully  funded  or  accomplished  in  a  reasonable  timeframe.  Thus,  some 
prioritization  must  be  made  among  the  various  alternatives. 

It  is  quite  possible  that  the  on-going  R  &  D  of  the  ether  TDS  subsystems 
may  provide  some  answers  relevant  to  these  TCS  research  issues,  or  at 
least  provide  data  useful  in  prioritizing  among  the  various  directions 
which  future  R  &  D  might  take.  In  addition,  some  of  the  issues  are  being 
addressed  in  the  continuation  phase  of  the  present  contract  (Ammendment 
No.  3).  For  the  purposes  of  this  report,  it  is  sufficient  to  conclude  that 
the  present  system  developed  for  the  prototype  TDS  works  reasonably  well 
even  though  additional  R  &  D  is  desirable  to  make  the  operation  of  the  TCS 
more  efficient  and  thus  more  cost-effective. 
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ACRONYMS  AND  ABBREVIATIONS 


AF  Air  Force 

AFB  Air  Force  Base 

AFS  Air  Force  Specialty 

ATC  Air  Training  Ccranand 

CDC  Career  Development  Course 

OODAP  Comprehensive  Occupational  Data  Analysis  Programs 

DIAGRM  Program  in  OODAP  for  displaying  hierarchical  clustering 
results 

FTD  Field  Training  Detachment 

FUS  Field  Utilization  Subsystem 

GRCXJP  Program  in  OODAP  for  clustering  cases  or  tasks 

IOS  Integration  and  Optimization  Subsystem 

KPATH  Clustered  sequence  (or  path)  for  cases  from  GROUP 

COT  On-the-Job  Training 

OS  Occupational  Survey 

OVRIAP  OODAP  program  to  compute  a  similarity  matrix 

FMI  Preventive  Maintenance  Inspection 

FRTVAR  Print  Variable  program  to  display  variable  values 

RCS  Resource/Cost  Subsystem 

SME  Subject-Matter  Expert 

SOW  Statement  of  Work 

STS  Specialty  Training  Standard 

TCS  Task  Characteristics  Subsystem 

TDS  Training  Decisions  System 

TPATH  Task  clustering  sequence  from  GROUP  with  transposed 
case  data 

TIM  Task  Training  Module 

U&T  Utilization  and  Training 

XPOSE  Program  in  COCAP  for  transposing  case  data  files 
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35 


Guidelines  fear  Constructing  Task  Training  Modules  (TIMs) 


This  Appendix  provides  general  guidelines  for  the  development  of  Task 
Training  Modules  (TIMs)  for  use  in  the  Training  Decisions  System  (TD6)  . 
These  recommended  procedures  are  based  on  the  research  and  development 
(R&D)  of  the  Task  Characteristics  Subsystem  (TCS)  in  the  prototype  TDS 
project  under  AFHRL  Contract  Number  F33615-83-C-0028.  This  TCS  R&D  is 
described  in  the  main  body  of  this  report  as  well  as  earlier  TCS-related 
contract  deliverables  (see  Perrin,  Vaughan  et  al.,  1986;  Perrin,  Mitchell, 
&  Knight,  1986) .  Specific  OODAP  runstreams  required  for  the  initial  phase 
of  this  procedure  are  detailed  in  the  Task  Characteristics  Subsystem 
Software  report,  delivered  to  AFHRL  in  June  1987. 

Overview  -  The  recommended  TIM  development  procedure  is  a  three-phased 
process: 

1.  Statistical  clustering  of  the  Occupational  Survey  (OS)  tasks 

for  the  specialty  using  co-performance  as  the  similarity 
measure; 

2.  Interpreting  the  co-performance  cluster  diagram,  identifying 

initial  TIMs  based  on  patterns  of  between  and  within 
group  similarity;  and, 

3.  Having  subject-matter  experts  (SMEs)  of  the  specialty  name  the 

task  groupings,  place  tasks  that  could  not  be  grouped  based 
on  the  clustering  results,  and  refine  the  initial  clusters 
to  form  final  TIMs. 

Guidelines  for  constructing  TIMs  are  organized  around  these  three 
steps,  and  details  are  provided  on  hew  each  step  should  be  coiiducLeJ. 
Such  guidelines  are  based  on  experience  with  the  four  specialties  studied 
in  the  prototype  TDS  project,  and  are  subject  to  possible  modification  as 
the  system  is  used  with  additional  Air  Force  specialties.  In  addition,  as 
more  of  the  new  ASCII  CODAP  tools  for  task  clustering  analysis  are 
operationally  tested  and  become  available  (Fhalen  et  al.,  1987),  additional 
guidelines  may  be  needed. 

PHASE  1.  -  STATISTICAL  CLUSTERING 

TTM  construction  begins  with  the  U.S.  Air  Force  Occupational 
Measurement  Center  (USAFOMC)  occupational  survey  (OS)  tasks  for  the 
specialty  under  study.  The  tasks  of  the  AFS  are  clustered  statistically 
using  co-performance  as  the  similarity  measure  to  generate  tentative 
TIMs.  This  process  involves  using  ASCII  OODAP  (developed  for  AFHRL  by  the 
MAXIMA  Corporation  under  Air  Force  Contract  F41800-87-  M8657)  files  and 
programs  in  a  different  way  (see  Figure  A-l) . 

A.  CONVERT  (if  necessary)  -  The  process  begins  with  existing  CODAP 
files  for  the  specialty  which  are  available  at  AFHRL  and  can  be  loaded  to 
the  AFHRL  Sperry-Univac  computer .  If  such  AFS  OODAP  files  are  recent 
(1987  or  later) ,  they  are  probably  in  ASCII  CODAP  format  and  can  be 
processed  directly.  For  older  USAFCMC  studies,  the  AFS  OODAP  file 
probably  requires  conversion  to  the  newer  ASCII  OODAP  format  to  create  the 
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Task  Clustering 


Statistical  Clustering  of  Tasks  to  Develop  Task  Training 
Modules.  (Flow  diagram  adapted  from  the  ASCII  CODAP 
Training  Package  developed  by  the  MAXIMA  Corporation  under 
AF  Contract  F41800-87-M8657) . 
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study  file.  For  task  clustering,  the  study  file  must  include  both  the 
Case  Data  File  and  Tcisk  Title  File.  (In  scene  studies,  if  you  wish  to 
display  any  task  factor  data,  the  Task  Factor  File  should  also  be 
included.) 

[The  conversion  process  is  documented  in  ASCII  OODAP  program  descriptions 
under  the  title  CONVERT.  To  review  this  program  description  (or 
descriptions  of  subsequent  programs)  on  the  AFHRL  Sperry-Univac ,  log-on 
with  a  user  ID,  then  enter: 

@Z*ZA.doc  - >  MENU 

Select  the  ASCII  CODAP  option  - >  list  doc  names 

Identify  CONVERT  program  number  (#) 

Enter  V  #  to  view  or  P  #  to  print  a  copy] 

B.  TRANSPOSE  -  Given  an  ASCII  OODAP  Case  Data  File,  the  first  step  in 
producing  a  task  co-performance  cluster  diagram  is  to  transpose  the  case 
data  using  an  ASCII  OODAP  program  titled  XPGSE.  Use  the  "raw  task 
responses"  option  in  transposing  the  file.  This  routine  changes  the  file 
from  a  case-oriented  to  a  task-oriented  form.  Thus,  rather  than  task 
performance  data  for  each  respondent,  the  resulting  new  file  is  composed 
of  respondent  performance  data  for  each  task.  Once  the  OS  Case  Data  File 
is  transposed,  clustering  is  performed  using  the  normal  CODAP  clustering 
sequence  of  OVRIAP,  GROUP,  and  DIAGFM. 

C.  OVRIAP  -  This  program  computes  a  similarity  measure  between  each 
pair  of  tasks  contained  in  the  transposed  Case  Data  File.  Select  the  co¬ 
performance  option  in  executing  this  program.  The  resulting  similarity 
matrix  computed  by  OVRIAP  includes  the  degree  to  which  all  pairs  (or 
groups)  of  tasks  are  jointly  performed  (average  co-performance)  for  all 
respondents  (cases) . 

[OVRIAP  will  normally  cluster  a  maximum  of  7,000  cases  and  3,000  tasks. 
With  a  transposed  case  data  set,  this  means  there  is  a  maximum  of  3,000 
cases  which  can  be  compared  at  one  time.  For  larger  Occupational  Survey 
Report  (OSR)  studies,  with  a  sample  size  greater  than  3,000,  a  random 
sampling  of  not  more  than  3,000  cases  should  be  used.  For  very  large  or 
important  studies,  multiple  random  samples  could  be  generated  and 
resulting  groupings  compared  to  ensure  the  generality  of  results.] 

D.  GROUP  -  This  program  reads  the  similarity  matrix  produced  by  OVRIAP 
and  organizes  the  most  similar  groups  into  hierarchical  clusters 
iteratively  until  all  tasks  have  been  included.  Default  options  are  used 
so  that  the  most  similiar  pair  (or  group)  is  selected  to  merge  and  their 
values  averaged  to  generate  values  for  the  new  group.  The  Output  Cluster 
File  records  the  collapse  of  tasks  into  groups  from  single  pairs  to  their 
merger  in  the  final  total  sample  stage  (GRP001) .  Another  product  of  this 
program  is  a  listing  of  tasks  in  cluster  sequence. 

[The  cluster  sequence  file,  formerly  known  as  KPATH,  is  an  important 
product  needed  for  the  interpretation  of  the  groups  and  clusters.  For 
task  data,  this  sequence  listing  is  sometimes  referred  to  as  a  TPATH  list 
or  order  (as  opposed  to  Case  Data  File  or  KPATH) .  Guch  a  TPATH  listing  is 


produced  by  requesting  a  FKIVAR  which  includes  the  task  statements  and 
task  identifiers  (task  number  and  duty  area)  for  use  during  analysis. 
ASCII  OODAP  also  includes  an  MPATH  program  option  which  gives  the  user  the 
ability  to  reorder  the  Cluster  sequence  based  on  certain  specified 
variables,  without  changing  the  results  of  the  clustering  process.  Thus, 
MPATH  changes  the  diagram  display  by  placing  higher  weighted  groups  or 
clusters  to  the  left,  giving  an  ordered  sequence.  This  option  is  not  used 
in  present  TDS  Task  Clustering  projects,  but  may  beccme  important  in 
future  Task  Clustering  developments  (for  example,  possible  nonhierarchical 
reclustering  refinement) . ] 

E.  DIAGRM  -  Ihis  program  prints  a  picture  which  shews  how  the  groups 
in  the  clustering  merged  together  to  form  a  single  composite  stage.  Ihe 
resulting  report  is  a  multipanel  tree  structure  which  displays  how  all  the 
tasks  of  the  specialty  relate  to  each  other  in  terms  of  average  co¬ 
performance.  This  product  is  the  starting  point  for  analysis  or 
interpretation  of  potential  task  groupings. 

PHASE  2.  ANALYSIS  OF  TASK  CLUSTERS 

Once  a  task  cluster  diagram  has  been  produced,  an  analyst  is  needed  to 
identify  potentially  meaningful  groups  of  tasks  as  potential  TIMs.  The 
analyst  needs  to  be  familiar  with  CODAP  hierarchical  diagrams  and  must  be 
able  to  assess  the  data  such  diagrams  provide. 

Since  task  cluster  diagrams  are  new  and  their  information  somewhat 
different  from  normal  case  cluster  diagrams,  seme  very  basic  details  will 
be  reviewed.  Figure  A-2  illustrates  an  element  of  a  hypothetical  task 
clustering  diagram  to  serve  as  the  basis  for  this  review. 

Stage  Number  -  This  is  the  identification  number  of  the  Group 

from  the  GROUP  program.  It  is  used  to  track  the 
group  through  various  programs  and  displays. 

In  a  typical  OS,  such  groups  would  be  com¬ 
posed  of  people  (cases) ;  in  this  situation,  the 
groups  are  sets  of  tasks. 

Number  of  Tasks  -  Literally,  the  number  of  tasks  in  the  group. 

TPATH  Sequence  -  The  range  of  sequence  numbers  identifying  which 
tasks  are  involved  in  this  group.  Cross-reference 
to  the  TPATH  listing  (ERTVAR)  . 

Overlap  Between  -  The  data  in  the  lower  left-hand  comer  of  the 
diagram  element  is  an  index  of  the  similarity  of 
the  two  groups  of  tasks  merging  to  form  this  new 
group.  It  represents  the  average  of  their  co¬ 
performance  values. 

Overlap  Within  -  once  the  new  group  has  been  formed,  the  average 
co-performance  among  the  tasks  is  reccrputed  to 
assess  internal  homogeneity  of  the  group;  this 
value  is  displayed  in  the  lower  right  comer  of 
the  diagram  element. 
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Figure  A- 2 .  Data  in  a  Typical  Task  Clustering  Diagram  Element. 


These  types  of  information  are  provided  for  each  stage  of  the  hier¬ 
archical  clustering  process.  Since  a  display  of  all  groups  would  be 
extremely  massive,  normally  only  a  portion  of  the  data  is  displayed  in  the 
diagram.  Typically,  GSR  diagrams  are  limited  to  "starter"  groups  of  5  or 
more,  or  with  overlap  values  less  than  some  specified  value.  For  Task 
Cluster  diagrams,  starter  group  size  should  be  set  lower  (at  2  or  3)  to 
permit  examination  of  pairs  or  triads  of  tasks. 

To  facilitate  further  discussion  of  identifying  meaningful  groups  of 
tasks,  a  portion  of  the  AFS  328X4  task  clustering  diagram  is  shown  as 
Figure  A-3.  Figure  A-4  displays  the  relevant  portion  of  a  PRTVAR  for  AFS 
328X4  in  cluster  sequence  (TPATH)  order  of  tasks. 

Using  these  two  ASCII  CODAP  products,  the  Task  Cluster  Diagram  and  the 
TPATH  Sequence  Listing,  an  analyst  must  identify  potentially  meaningful 
groups  of  tasks  to  be  tentative  TIMs.  This  is  accomplished  by  systematic 
examination  of  portions  of  both  documents  for  patterns  of  variance. 

A.  Diagram  -  A  visual  inspection  of  the  diagram  (Figure  A-3)  indicates 
generally  high  Between  and  Within  overlap  values.  This  appears  to  be 
typical  of  those  parts  of  a  task  list  involving  specialized  technical 
areas  (specific  equipment  maintenance,  procedures) ,  with  much  lower 
overlap  values  for  managerial  and  supervisory  tasks.  Clearly,  however, 
the  usual  rules -of-thumb  for  OS  (case)  diagrams,  such  as  the  35/50  rule, 
do  not  apply.  Application  of  such  a  rule  would  result  in  extremely  large 
TIMs  of  75  to  150  tasks  going  across  several  duty  areas  and  involving 
several  equipment  systems.  Rather,  the  analyst  must  examine  the  relative 
overlap  values  in  the  vertical  linkages  of  the  diagram,  paying  particular 
attention  to  groups  where  there  is  substantial  drop-off  in  overlap  as 
groups  merge. 
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Example  Task  Cluster  Diagram  (AES  328X4) . 
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Sequence 

No.  or 

TPATH 

Duty 

Task 

Task  Title 

0708 

w 

651 

Inspect  Clean  Air  Stations 

0709 

W 

695 

Isolate  Malfunctions  to  Clean  Air  Station  Coirponents 

0710 

W 

758 

Repair  Clean  Air  Stations 

0711 

J 

322 

Remove  or  Reel ace  Headina  Couriers 

0712 

T 

546 

Isolate  Malfunctions  to  MADAR  Computers 

0713 

T 

547 

Isolate  Malfunctions  to  MADAR  Control  &  Sequence  Units 

0714 

T 

548 

Isolate  Malfunctions  to  MADAR  Data  Retrieval  Units 

0727 

T 

567 

Remove  or  Replace  MADAR  SAR 

0728 

T 

560 

Remove  or  Replace  MADAR  SCU 

0729 

T 

563 

Remove  or  Replace  MADAR  MMUX 

0730 

T 

565 

Remove  or  Replace  MADAR  POU 

0731 

T 

562 

Remove  or  Replace  MADAR  MDR 

0732 

T 

568 

Remove  or  Replace  MADAR  SCA  Units,  such  as 

Temperature,  Pressure,  or  Vibration 

0733 

T 

570 

Reprogram  MADAR  Computers 

0734 

T 

569 

Remove  or  Replace  MADAR  SCM 

0735 

T 

566 

Remove  or  Replace  MADAR  Recording  Tapes 

0736 

T 

557 

Operate  MADAR  for  System  Interface  Check-outs 

0737__ 

_T_ 

571 

Teach  MADAR  Troubleshooti_ng_Techniques 

0738 

U 

572 

Adjust  MADAR  CMA 

0739 

U 

576 

Adjust  MADAR  MMUX 

0740 

U 

573 

Adjust  MADAR  SCU 

0741 

U 

577 

Adjust  MADAR  ODRJ 

0742 

U 

575 

Adjust  MADAR  MDR 

0743 

U 

574 

Adjust  MADAR  DKU 

0744 

U 

578 

Align  MADAR  CMA 

0745 

U 

579 

Align  MADAR  CSU 

0746 

U 

581 

Align  MADAR  MDR 

0771 

u 

608 

Program  MADAR  Computers 

0772 

u 

596 

Bench  Check  MADAR  SCM 

0773 

u 

606 

Isolate  Malfunctions  to  SCA  SRU 

0774 

u 

607 

Isolate  Malfunctions  to  SCM  SRU 

0775 

K 

347 

Align  Heading  Couplers 

0776 

K 

358 

Bench  Check  Heading  Couplers 

0777 

K 

372 

Isolate  Malfunctions  to  Heading  Coupler  SRU 

0778 

K 

385 

Remove  or  Replace  Heading  Coupler  SRU 

Figure  A-4.  Example  of  TPATH  Sequence  Task  Listing  (Extracted  from  "PETVAR 
for  AFS  328X4  Tasks  in  KPATH  Order,  Transpose  of  Raw  Data") . 
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EXAMPLE:  For  the  AFS  328X4  diagram,  good  candidate  TIMs  would  be 
(reading  up  from  the  bottom  of  the  figure)  groups  0219, 
0691,  0747,  and  the  pairs  of  groups  merging  to  form  the 
latter  two  groups  (0695  &  0704;  0527  &  0536).  Note  that 
the  Between  and  Within  overlap  for  GRP  0691  (96.4  &  97.3) 
drops  substantially  when  it  merges  with  GRP  0747  to  form 
GRP  0219  (down  to  63.6  and  77.5),  where  in  the  interven¬ 
ing  steps  (0268  &  0529),  the  reduction  is  more  gradual. 
We  need  to  examine  the  tasks  involved  (TPATH  735,  736, 
737,  738,  etc.)  to  see  if  we  can  see  hew  these  groups 
are  different  (and  hew  they  are  the  same) . 


B.  Cluster  Sequence  (TPATH)  -  An  analyst  needs  to  also  examine  the 
listing  of  tasks  in  clustered  (TPATH)  sequence  (see  example  in  Figure  A-4) 
again  looking  for  patterns  of  variance.  In  this  case,  one  should  examine 
Duty  and  Task  identifiers  for  strings  or  patterns  of  related  tasks,  as  well 
as  the  content  of  the  task  titles  or  statements  themselves  for  common 
content  (equipment,  procedures,  action  verbs,  etc.). 

EXAMPLE:  A  look  at  the  AFS  328X4  FKTVAR  indicates  that  all  the 
tasks  in  this  part  of  the  diagram  (TPATH  0712  -  0774) 
involve  MADAR  maintenance  (OSR  Duties  T  and  U) .  There 
is  a  break  after  TPATH  0774  (where  GRP  0219  ends)  with 
a  switch  to  Duty  K;  these  Duty  K  tasks  involve  Heading 
Couplers,  which  is  quite  a  different  system.  Even  with¬ 
in  TPATH  sequence  0712  -  CC/4,  there  is  a  distinct  break 
between  tasks  0737  (which  is  a  IXity  T  task)  and  0738 
(which  is  Duty  U) .  The  IXity  T  tasks  are  Remove  and  Re¬ 
place  tasks  whereas  tne  IXity  U  tasks  involve  Adjusting, 
Aligning,  and  Bench  Checking  MADAR  components.  This 
appears  a  meaningful  difference  (Overlap  drop-off,  IXity 
and  content  difference) .  This  difference  seems  to  be 
associated  with  flightline  (on-aircraft)  maintenance 
(the  remove  and  replace  tasks)  and  shop  (off-aircraft) 
maintenance  (adjust,  align,  bench  check,  etc.).  This 
difference  is  striking  in  this  example,  but  in  other 
parts  of  the  diagram  may  be  mixed  to  a  greater  degree. 

The  only  difference  in  this  sequence  and  GRP  0219  on  the 
diagram  is  Task  0711  (J  322  Remove  or  Replace  Heading 
Couplers) ;  it  does  not  fit  into  the  MADAR  repair  group. 

C.  Isolate  Tasks  -  Once  the  potential  TIMs  are  identified,  there  still 
remain  some  tasks  which  are  not  included,  or  which  appear  inappropr iate ; 
some  may  be  related  to  the  overall  area,  but  enter  the  clustering  lower  down 
on  the  diagram  after  the  major  clusters  have  formed.  These  tasks  are  termed 
"Isolates"  since  they  are  not  part  of  the  identified  groups;  decisions  must 
be  made  as  to  how  to  handle  such  tasks.  Are  they  a  group  or  should  they  be 
considered  isolated  individual  tasks?  Should  they  be  forced  into  one  of  the 
major  groupings  and,  if  so,  which  one?  Are  they  related  in  content  to  this 
area  of  the  diagram,  or  to  some  other  region?  If  a  quick  look  at  such  tasks 
indicates  they  have  some  content  relationship  to  this  area,  they  should  be 
dealt  with  here;  if  not,  defer  their  consideration  until  all  major  areas 
of  the  diagram  have  been  evaluated. 
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EXAMPLE:  Task  0711  (J  322  Remove  or  Replace  Heading  Couplers)  is 
the  only  J  task  in  the  example  TPATH  listing — it  simply 
does  not  fit.  In  content,  it  appears  more  like  the  Duty 
K  tasks  (TPATH  sequence  775  -  778)  at  the  end  of  the 
list;  those  tasks  also  involve  Heading  Couplers. 
Typically,  an  analyst  will  not  have  sufficient  knowledge 
of  the  technical  area  involved  to  determine  if  Task  711 
should  be  arbitrarily  moved  to  be  a  part  of  a  Heading 
Coupler  TIM,  or  should  be  considered  a  separate  TIM  in 
its  own  right.  In  addition,  two  of  the  tasks  at  the 
interface  of  the  diagram  groups  (TPATH  736  and  737)  also 
are  qualitatively  different  (Operate  and  Teach  rather 
than  maintain) .  Note  that  on  the  Diagram  (Figure  A-3) 
these  two  tasks  enter  below  GRP  691;  that  is,  they  are 
outside  of  the  two  major  MADAR  maintenance  TTMs.  These 
tasks  may  be  performed  by  everyone  involved  with  MADAR 
or  by  only  a  few  individuals.  For  resolving  this  kind 
of  evaluation  problem,  the  judgment  of  SMEs  is  of 
critical  importance. 

In  general,  an  analyst  can  select  a  reasonable  set  of  tentative  TTMs 
using  the  procedures  outlined  above.  In  some  cases  (as  in  our  MADAR 
example) ,  the  common  theme  of  the  tasks  involved  clearly  suggests  the 
generic  principle  for  the  task  groupings  (such  as  On-Aircraft  MADAR 
Maintenance  versus  MADAR  Shop  Maintenance) .  There  remain,  however,  some 
isolated  tasks,  or  sets  of  tasks,  which  are  ambiguous  in  their  relation¬ 
ship  to  other  task  groups.  Typically,  at  the  end  of  the  TPATH  sequence 
list,  there  will  also  be  a  few  tasks  which  do  not  appear  to  fit  anywhere; 
they  may  be  individual  TTMs  in  their  own  right,  or  may  in  fact  be  related 
to  seme  identified  group  in  seme  way  not  obvious  to  the  analyst. 

PHASE  3.  -  SUBJECT-MATTER  EXPERT  REVIEW 

A  review  of  the  tentative  TTMs  (selected  by  an  analyst)  by  a  represen¬ 
tative  group  of  technical  experts  who  are  thoroughly  familiar  with  the  Air 
Force  specialty  involved  serves  a  number  of  purposes.  These  SMEs  can 
quickly  validate  the  groupings  selected  by  the  analyst.  Secondly,  SMEs 
are  more  technically  qualified  to  decide  a  proper  name  for  the  TIMs;  their 
generic  term  for  each  TIM  should  communicate  the  basic  content  of  each  TIM 
to  anyone  in  the  specialty.  Thirdly,  SMEs  can  quickly  resolve  the 
questions  an  analyst  has  about  hew  Isolates  fit  into  the  work  of  the 
specialty  and  to  which  TIM  (or  TTMs)  such  Isolates  relate.  Finally,  for 
those  anomalous  tasks  which  do  not  fit  anywhere,  SMEs  can  make  realistic 
judgments  as  to  whether  such  tasks  should  be  treated  as  individual  TIMs, 
or  merely  be  left  as  a  "Miscellaneous"  TIM. 

Ideally,  a  representative  panel  of  SMEs  should  be  convened  to  conduct 
this  TIM  validation.  In  practice,  this  may  be  impossible  or  impractical 
due  to  the  wide  geographic  distribution  of  specialized  Air  Force  jobs.  In 
any  case,  the  TIM  review  must  be  done  in  a  systematic  way  both  to  ensure 
that  good  judgments  are  made  and  to  minimize  the  amount  of  time  SMEs  must 
be  diverted  from  their  operational  duties. 

A.  Construction  of  TTM  Review  Booklet  -  The  analyst  develops  a  special 
booklet  of  TTMs  for  use  in  SME  reviews.  The  ASCII  CODAP  program  called 


MODULES  can  be  used  to  construct  this  booklet;  the  analyst  maps  the  tasks 
into  numbered  modules  (without  narrative  titles,  for  the  present) .  The 
final  module  should  include  all  tasks  not  grouped,  to  ensure  that  all 
tasks  are  considered. 

[Since  manipulation  of  tasks  out  of  their  normal  Duty  and  Task  Number 
sequence  can  be  complex,  it  is  good  to  have  the  program  count  the  number 
of  task  entries.  This  total  should  equal  the  number  of  tasks  in  the  USAF 
Job  Inventory;  if  not,  then  some  task  has  been  omitted  or  included  more 
than  once. ] 

The  analyst  then  requests  a  printout  in  module  order.  Task  iden¬ 
tifiers  (Duty,  Task  No.,  TPATH  No.)  should  be  omitted  to  focus  the  SME 
review  on  task  content.  Have  each  TIM  printed  on  a  separate  page;  again, 
this  format  tends  to  focus  SME  review  on  the  commonality  of  the  tasks  on  a 
page  and  thus  expedites  their  review  and  validation  process.  The  printout 
should  have  a  title  page  which  includes  the  AFSC  involved  and  notes  that 
these  are  "TENTATIVE  Task  Training  Modules." 

B.  Selecting  Representative  SMEs  -  The  diversity  of  jobs  and  tasks 
within  an  Air  Force  specialty  dictates  the  number  and  qualifications  of 
SMEs  needed  for  TTM  validation.  For  a  typical  specialty  where  most  tasks 
are  performed  by  most  individuals,  a  group  of  three  or  four  SMEs,  prefer¬ 
ably  TSgt  or  MSgts  who  have  been  assigned  to  a  number  of  bases,  would 
suffice.  For  more  complex  specialties,  with  very  diverse  and  specialized 
jobs,  the  number  of  SMEs  required  will  be  greater.  The  CSR  provides 
general  information  about  the  diversity  of  jobs  and  the  MAJCOM 
distribution  of  personnel  in  the  specialty.  It  does  not  typically  include 
information  as  to  specific  bases  where  identified  jobs  are  performed. 

[The  Background  Data  FRTVAR  for  the  OSR  study  should  be  retrieved  from 
tape  storage  at  AFHRL  and  printed,  to  provide  a  good  basis  for  SME 
sampling.  Include  KPATH  sequence  numbers  in  the  product  to  be  printed, 
along  with  name,  AFS,  base,  etc.;  the  KPATH  numbers  are  needed  to  link 
individuals  to  specific  job  types,  and  thus,  indirectly,  to  identify  where 
the  groups  of  tasks  are  performed.] 

C.  Group  Interviewing  or  Surveying  -  If  sufficient  representative 
SMEs  can  be  identified  in  the  local  area,  a  group  meeting  should  be 
convened  to  validate  and  review  the  TIMs.  If  this  is  not  possible, 
because  of  operational  commitments  or  lack  of  travel  funds,  then  a  mail 
survey  may  be  the  only  option. 

In  either  case,  the  SMEs  should  be  asked  to  first  read  through  the  TTM 
Review  booklet  completely  to  orient  themselves  to  the  task  statements  and 
tentative  groupings.  They  should  be  instructed  to  go  through  the  booklet 
a  second  time  and  name  each  TTM  and  closely  examine  the  tasks  of  each  TIM 
to  ensure  that  all  tasks  are  appropriate  for  this  grouping.  They  should 
also  be  instructed  to  carefully  examine  the  final  page,  the  ungrouped 
tasks,  and  decide  whether  these  tasks  should  be  placed  with  seme  TIM,  made 
a  separate  TTM,  or  left  in  a  Misc.  category.  If  the  review  is  conducted 
as  a  group  session,  SMEs  should  compare  their  decisions  and  arrive  at  a 
consensus  judgment  for  each  of  the  issues.  For  mailed  surveys,  the 
analyst  must  consolidate  responses  and  make  some  arbitrary  decisions  on 
the  most  appropriate  titles  and  possible  changes  to  TIMs  or  the  ungrouped 
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tasks.  Such  changes  cculd  be  discussed  by  phone  with  survey  participants. 

D.  Preparing  Final  TIM  Listing  -  A  final  step  in  the  TIM  Construction 
process  is  to  enter  the  names  or  titles  selected  for  TIMS  into  the  TIM 
Module  File  and  make  the  other  changes  decided  by  the  SMEs  (or,  in  the 
case  of  mail  review,  by  the  analyst) ,  such  as  any  shifting  of  tasks  or 
creation  of  new  TIMS.  By  using  the  MODULES  programs  for  constructing  the 
TIM  Review  booklets,  the  identity  of  the  TIMs  and  their  task  content  are 
automatically  part  of  the  ASCII  OODAP  AFS  study  file.  Ihe  revised  file  is 
then  available  for  use  in  other  TDS  activities  and  serves  as  the 
foundation  for  the  TDS  study  of  the  Air  Force  specialty  involved. 
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Sample  of  the  first  of  two  TIM  Allocation  Survey 
formats  field-tested  in  AFS  811XX  at  Lackland  AFB,  TX 
(see  section  3.1  for  a  description  of  this  field  test) . 
Included  are  the  instructions,  one  of  the  task  modules 
evaluated,  and  the  response  sheet  used  to  record  alloca¬ 
tion  decisions.  Respondents  used  a  "percent  of  full 
training"  measure  to  describe  different  allocations. 


GENERAL  INSTRUCTIONS 


Your  assistance  in  completing  this  survey  is  very 
important  to  the  Air  Force.  AF  training  may  occur  in 
several  different  settings  including  resident  technical 
schools  (CLASS),  Career  Development  Courses  (CDC) , 
Educational  Subject  Block  Indices  (ESBI>,  and  on-the-job 
training (OJT) .  One  important  decision  in  building  effective 
training  programs  is  how  to  divide  training  among  these 
training  settings  (how  much  to  teach  where). 

This  questi onnai re  is  designed  to  determine  the 
following  for  a  group  of  occupational  tasks: 

1)  the  LEVEL  OF  TRAINING  that  is,  should,  and  can  be 
provided  in  each  of  the  training  settings;  and 

2)  the  TIME  it  will  take  to  train  the  tasks  to  these 
levels  in  each  setting. 


KEY  CONCEPTS 


Training  Settings 

CLASS:  All  resident  technical  training  including  basic 

and  advanced  courses  relevant  to  the  tasks  (tech  schools). 
All  Professional  Military  Education  (PME)  that  occurs  in  the 
classroom  should  be  included  in  this  category. 

CDC:  All  Career  Development  Courses  relevant  to  the 

tasks.  All  management  training  by  corr espondence  should  be 
included  in  this  category. 

ESBI:  All  training  provided  through  Educational  Subject 

Block  Indices  relevant  to  the  tasks. 

OJT:  All  activities  of  the  trainee  and  the  trainer 

relevant  to  learning  the  tasks.  Relevant  trainee  activities 
may  include  practice,  studying  training  or  technical 
manuals,  receiving  instruction,  or  observing  others 
demonstrating  the  tasks.  Relevant  trainer  activities  may 
include  verifying,  monitoring,  and  observing  the  trainee’s 
work  and  providing  instruction,  direction,  and  feedback  on 
that  performance. 

Training  Time 

The  number  of  hours  required,  on  average,  for  a  trainee 
to  achieve  a  given  level  of  training  in  a  setting.  "eparate 
estimates  of  trainer  time  will  also  be  needed  for  OJT. 
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Level  of  Training 


The  level  o-f  training  in  a  setting  can  go  from  no 
training  to  full  training.  For  this  quest i onnai r e,  full 
training  means  "able  to  perform  the  tasks  to  minimum 
acceptable  standards  without  direct  supervision".  It  does 
NOT  mean  expert  or  highly  skilled  performance. 

To  describe  the  level  of  training  for  each  setting, 
indicate  the  percent  of  full  training  provided.  0%  means  no 
training  is  to  be  provided  in  a  setting.  100%  means  that 
all  training  necessary  for  an  airman  to  satisfactorily 
perform  the  tasks  without  supervision  is  to  be  provided  in 
that  setting. 

Current  Training  Level  and  Training  Time 

The  level  of  training  provided  in  each  setting  under  the 
current  training  system  and  the  time  it  takes  to  provide 
this  training  (the  way  things  are  now).  Note  that  the 
percents  of  full  training  must  add  to  1007.. 

EXAMPLE: 

Training  Time  (in  hours) - 

Level  of  Training-  ■■  ■  » 

(percent  of  full  training) 

v 

Current  Training  Level  CLASS;  '3o7o 

(the  way  things  are  now)  E5B I :  -2-0*7.. 

CPC:  C7. 

OJT :  s~C7o 


TOTAL:  1007. 

Ideal  Training  Level  and  Training  Time 

Level  of  training  that  should  be  provided  to  make  the 
best  use  of  a  training  setting,  DISREGARDING  costs  and 
number  and  location  of  airmen  performing  the  tasks  (the  way 
things  should  be).  Again,  the  percents  must  add  to  1007.. 

EXAMPLE: 

Ideal  Training  Level  CLASS:  3C70  1 Q 

(the  way  things  should  be)  ESB 1 :  307«»  2 

CPC:  07.  - 

0.1  T «  *f07,  iS  (trainee) 

XP  ( trainer ) 

TOTAL:  1007. 


V 

IQ _ 

Jb _ 

XQ  ( trai  nee ) 
XS  (trainer ) 
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Minimum  Entering  Background 

Minimum  level  of  training  (if  any)  an  airman  should  have 
before  beginning  training  in  a  setting,  so  that  this 
training  may  be  effective.  Note  that  the  minimum  training 
level  must  be  OX  for  at  least  one  setting;  these  are 
settings  which  require  no  previous  training.  Percents  need 
not  add  to  100X  and  no  training  times  are  needed. 

EXAMPLE: 

Minimum  Entering  Background  CLASS:  O 

ESB I :  QVo 
CPC:  iO-Iq 
OJTi  2-01» 


Maximum  Training  Level  and  Training  Time 

Maximum  level  of  training  that  can  be  reached 
effectively  in  a  setting.  Percents  need  not  add  to  100X. 
Training  times  indicate  the  time  required  to  reach  the 
maximum  level  if  the  airman  started  with  the  minimum 
entering  background. 

EXAMPLE 


Maximum  Training  Level 


CLASS: 

50^0 

2.2. 

ESBI: 

2-5 

CDC: 

3on0 

2.0 

OJT: 

!OC7o 

.LL. 

^0  (trainer ) 


INSTRUCTIONS 

On  the  next  page,  you  will  find  some  tasks  that  have 
been  grouped  together  because  they  are  coperformed;  that  is, 
i f  an  airman  does  one  of  the  tasks,  he  or  she  probably  does 
one  or  more  of  the  others.  Read  through  the  list  of  tasks. 
Then,  using  the  percent  of  full  training  provided,  describe 
the  current  training  level,  the  ideal  training  level,  the 
minimum  entering  background,  and  the  maximum  training  level 
for  each  training  setting.  Also  indicate  the  training  time 
needed  to  achieve  these  levels  in  each  setting. 
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TASK  GROUP  NO.  1 


Processing  Incident  Investigation  Forms 


Make  entries  on  AF  Forms  1168/1170  (Statement  of 
Suspect ) 

Make  entries  on  AF  Forms  1169/1170  (Statement  of 
Wi tness) 

Make  entries  on  AF  Forms  1176  (Authority  to  Search  and 
Seize) 

Make  entries  on  AF  Forms  1364  (Consent  for  Search  and 
Sei zure) 

Make  entries  on  AF  Forms  1668  (Field  Interview) 

Make  entries  on  AF  Forms  52  (Evidence  Tag) 

Concerning  the  above  tasks,  I  ams 
(mark  one  box) 

□  □  □  □ 

not  at  all  slightly  moderately  very 

familiar  familiar  familiar  familiar 

with  them  with  them  with  them  with  them 

If  you  marked  the  first  box,  please  skip  the  next  page  and 
continue  with  the  next  group  of  tasks. 


Minimum  Entering  Background  CLASS: _ 

ESBI: _ 

CDC: _ 

OJT : _ 

Maximum  Training  Level  CLASS: _  _ 

ESBI: _  _ 

CDC: _ 

□JT: _  _ (trainee) 

_ (trainer ) 
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Sample  of  the  second  of  two  TIM  Allocation  Survey 
formats  field-tested  in  APS  811XX  at  Lackland  AFB,  TX 
(see  section  3.1  for  a  description  of  this  field  test) . 
Included  are  the  instructions,  one  of  the  task  modules 
evaluated,  and  the  response  sheet  used  to  record  alloca¬ 
tion  decisions.  Respondents  used  a  seven-point  rating 
scale  to  describe  different  allocations. 


i 

1 


GENERAL  INSTRUCTIONS 


Your  assistance  in  completing  this  survey  is  very 
important  to  the  Air  Force.  AF  training  may  occur  in 
several  different  settings  including  resident  technical 
schools  (CLASS),  Career  Development  Courses  (CDC) , 
Educational  Subject  Block  Indices  (ESBI),  and  on— the— job 
trai ni ng (OJT) .  One  important  decision  in  building  effective 
training  programs  is  how  to  divide  training  among  these 
training  settings  (how*  much  to  teach  where). 

This  questi onnai r e  is  designed  to  determine  the 
following  for  a  group  of  occupational  tasks: 

1)  the  LEVEL  OF  PERFORMANCE  that  is,  should,  and  can 
be  attained  in  each  of  the  training  settings;  and 

2)  the  TIME  it  will  take  to  train  the  tasks  to  these 
levels  in  each  setting. 


KEY  CONCEPTS 


Training  Settings 

CLASS:  All  resident  technical  training  including  basic 

and  advanced  courses  relevant  to  the  tasks  (tech  schools). 

All  Professional  Military  Education  (PME)  that  occurs  in  the 
classroom  should  be  included  in  this  category. 

CDC:  All  Career  Development  Courses  relevant  to  the 

tasks.  All  management  training  by  correspondence  should  be 
included  in  this  category. 

ESBI:  All  training  provided  through  Educational  Subject 

Block  Indices  relevant  to  the  tasks. 

OJT:  All  activities  of  the  trainee  and  the  trainer 

relevant  to  learning  the  tasks.  Relevant  trainee  activities 
may  include  practice,  studying  training  or  technical 
manuals,  receiving  instruction,  or  observing  others 
demonstrating  the  tasks.  Relevant  trainer  activities  may 
include  verifying,  monitoring,  and  observing  the  trainee’s 
work  and  providing  instruction,  direction,  and  feedback  on 
that  performance. 

Training  Time 

The  number  of  hours  required,  on  average,  for  a  trainee 
to  achieve  a  given  level  of  training  in  a  setting.  Separate 
estimates  of  trainer  time  will  also  be  needed  for  OJT. 
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Level  of  Performance 


To  indicate  the  level  of  performance  attained  in  a 
setting,  please  use  the  following  scale. 

The  trainee  can  perform: 

0  =  no  aspects  of  the  tasks  independently;  requires 
constant  direction 

1  3  very  few  aspects  of  the  tasks  independently;  requires 

nearly  constant  direction 

2  2  some  aspects  of  the  tasks  independently;  requires 

substantial  direction 

3  3  many  aspects  of  the  tasks  independently;  requires  some 

direction 

4  =  most  aspects  of  the  tasks  independently;  requires 

little  direction 

5  3  nearly  all  aspects  of  the  tasks  independently; 

requires  very  little  direction 
b  -  all  aspects  of  the  tasks  to  minimum  standards  without 
direct  supervision. 

Performance  Level  Prior  to  Training 

The  performance  level  of  airmen  prior  to  any  AF 
training.  For  simple  tasks,  this  will  be  high;  for 
difficult  tasks,  this  will  be  low.  Note  that  no  training 
time  estimate  is  needed. 

EXAMPLE: 


Training  Time  (in  hours) 


Level  of  Training - 

(using  the  rating  scale) 


Performance  Level  Prior  to  Training: 


v 


Current  Training  Level  and  Training  Time 

The  level  of  performance  attained  in  each  setting  under 
the  current  training  system  and  the  time  it  takes  to  provide 
this  training  (the  way  things  are  now) i  Note  that  one 
setting  must  receive  a  rating  of  6,  indicating  that  minimum 
acceptable  performance  has  been  attained  from  the  training 
program. 


Current  Training  Level 

CLASS :_ 

Z 

IO 

(the  way  things  are  now) 

ESBI 

3 

to 

CDC:_ 

- 

- 

OJT 

b 

iO 

.  ( trainee) 

x5 

„  (trainer) 
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Ideal  Training  Level  and  Training  Time 

Level  of  performance  that  should  be  sought  to  make  the 
best  use  of  a  training  setting,  DISREGARDING  costs  and 
number  and  location  of  airmen  performing  the  tasks  (the  way 
things  should  be).  Again,  one  setting  must  receive  a  rating 
of  6. 

EXAMPLE: 


Ideal  Training  Level 

CLASS: 

X 

10 

(the  way  things  should  be) 

ESBI: 

8 

CDC: 

- 

OJT:. 

<o 

IS 

XO  ( t  r  a i ner ) 


Minimum  Entering  Background 

Minimum  level  of  performance  an  airman  should  have 
before  beginning  training  in  a  setting,  so  that  this 
training  may  be  effective.  No  training  times  are  needed. 

EXAMPLE: 

Minimum  Entering  Background  CLASS:  I 

ESBI :  I 
CPC:  X 
OJT :  3 


57 


Maximum  Training  Level  and  Training  Time 


Max i m^m  1 evel 
effectively  in  a 
required  to  reach 
with  the  minimum 

of  performance  that  can 
setting.  Training  times 
the  maximum  level  if  the 
entering  background. 

be  reached 
indicate  the  time 
airman  started 

EXAMPLE 

Maximum  Training 

Level 

CLASS > 

5 

xx 

ESBI : 

S' 

zS’ 

CDC: 

ZO 

OJT: 

lo 

3b  (trainee) 

MO  ( trai  ner ) 


INSTRUCTIONS 

On  the  next  page,  you  will  find  some  tasks  that  have 
been  grouped  together  because  they  are  coperformed;  that  is, 
if  an  airman  does  one  of  the  tasks,  he  or  she  probably  does 
one  or  more  of  the  others.  Read  through  the  list  of  tasks. 
Then,  using  the  rating  scale  for  the  level  of  performance, 
describe  the  current  training  level,  the  ideal  training 
level,  the  minimum  entering  background,  and  the  maximum 
training  level  for  each  training  setting.  Also  indicate  the 
training  time  needed  to  achieve  these  levels  in  each 
setting. 
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TASK  GROUP  NO.  1: 


Processing  Incident  Investigation  Forms 


Make  entries  on  AF  Forms  116S/1170  (Statement  of 
Suspect) 

Make  entries  on  AF  Forms  1169/1170  (Statement  of 
Mi tness ) 

Make  entries  on  AF  Forms  1176  (Authority  to  Search  and 
Sei ze) 

Make  entries  on  AF  Forms  1364  (Consent  -for  Search  and 
Sei zure) 

Make  entries  on  AF  Forms  166S  (Field  Interview) 

Make  entries  on  AF  Forms  52  (Evidence  Tag) 

Concerning  the  above  tasks,  I  am: 

(mark  one  box) 

□  □  □  □ 

not  at  all  slightly  moderately  very 

familiar  familiar  familiar  familiar 

with  them  with  them  with  them  with  them 

If  you  marked  the  first  box,  please  skip  the  next  page  and 
continue  with  the  next  group  of  tasks. 

Use  the  scale  below  to  describe  the  levels  of  training  on 
the  next  page. 

The  trainee  can  perform: 

0  -  no  aspects  of  the  tasks  independently;  requires 
constant  direction 

1  =  very  few  aspects  of  the  tasks  independently; 

requires  nearly  constant  direction 

2  =  some  aspects  of  the  tasks  independently;  requires 

substantial  direction 

3  =  many  aspects  of  the  tasks  independently;  requires 

some  direction 

4  =  most  aspects  of  the  tasks  independently;  requires 

little  direction 

5  =  nearly  all  aspects  of  the  tasks  independently; 

requires  very  little  direction 

6  =  al 1  aspects  of  the  tasks  to  minimum  standards 

without  direct  supervision. 
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Training  Time  (in  hours)- 


Level  of  Training - 

(percent  of  full  training) 


Current  Training  Level 

(the  way  things  are  now) 


CLASS: 


ESBI : 


TOTAL:  1007. 


Ideal  Training  Level  CLASS: 

(the  way  things  should  be) 

ESBI: 


TOTAL:  1007. 


(trainee) 


(tr ai ner ) 


(trai nee) 


(trainer ) 


Minimum  Entering  Background 


CLASS: 


ESBI : 


Maximum  Training  Level 


CLASS: 


ESBI: 


(trai nee) 


( trainer ) 
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Sample  of  the  final  TIM  Allocation  Survey  format. 

This  format  was  used  for  the  field  test  in  AFS  328X4 
at  Keesler  AFB,  MS  (see  section  3.1  for  a  description 
of  this  field  test).  Included  are  the  instructions, 
one  of  the  task  modules  evaluated,  and  the  response 
sheet  used  to  record  allocation  decisions.  Respondents 
used  "time  to  train"  to  describe  different  allocations. 
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PURPOSE 


The  purpose  of  this  questionnaire  is  to  describe  the  AMOUNTS  of 
different  types  of  training  that  are,  should,  or  could  be  provided  for  a 
group  of  tasks.  Training  may  be  through  classroom  Instruction, 
correspondence  courses,  hands-on  experience,  or  other  types  of 
instruction.  Training  efficiency  can  be  improved  if  the  proper  "mix"  of 
these  different  types  of  training  is  provided. 


INSTRUCTIONS — CURRENT  TRAINING 


For  each  type  of  training  described  in  the  questionnaire,  you  are  to 
indicate  the  TIME  you  believe  is  currently  devoted  to  reach  MINIMUM 
STANDARDS  for  each  group  of  tasks.  Remember,  this  is  training  to 
minimum  standards  only  and  does  NOT  mean  expert  or  highly  skilled 
performance.  In  terms  of  the  GO/NO  GO  concept,  this  is  training  up  to 
the  GO  level  only. 

Your  responses  should  describe  the  current  training  system  (the  way 
things  are  now)  as  accurately  as  you  can. 

The  example  on  the  next  page  shows  you  what  a  completed  questionnaire 
would  look  like. 
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EXAhPLE 


TASKS: 

Type  letter  from  rough  draft. 
Type  reports  from  rough  draft. 


For  each  type  of  training,  please  indicate  the  time  you  believe  is 
currently  devoted  to  reach  MINIMUM  STANDARDS  for  the  group  of  tasks 
above. 


Training 

Time 

Estimate 
(in  hrs) 


TYPE  OF  TRAINING 


Classroom  instruction  involving  lecture/discussion  _ 

A  related  reading  (most  resident  technical  training) 

Correspondence  courses;  self-paced,  individual  study  i 

from  text  (all  CDCs) 

Hands-on  experience  in  small,  supervised  training  c 

groups  using  simulators,  mock-ups,  or  actual 
equipment  (most  FTDs) 

Hands-on  experience  on-the-job  including  observing  -f 

others,  practicing  the  tasks,  A  receiving  direction 
(qualification  A  upgrade  training) 

Other,  please  specify: _  _ 


Please  turn  to  the  next  page  and  begin. 
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TASK  MODULE  24:  PEPFORM  AMD  DOCUMENT  NAVIGATION  EQUIPMENT  MAINTENANCE  ON  AIP 

LOCATE  PARTS  OP  STOCK  NUMBERS  IN  TECHNICAL  PUBLICATIONS 

RESEARCH  OR  IDENTIFY  PARTS  USING  ILLUSTRATED  PAPTS 
BREAKDOWNS  <  I P  8 ) 

MAKE  ENTRIES  ON  AFTO  FORMS  349  (MAINTENANCE  DATA  COLLECTION 
RECOPD) 

MAKE  ENTRIES  ON  AFTO  FORMS  350  (REPARABLE  ITEM  PROCESSING 
TAG) 

OPERATE  INERTIAL  AND  RADAR  NAVAGAT I  ON AL  EQUIPMENT  FOR 
CHECK-OUT  OF  ASSOCIATED  AVIONIC  SYSTEMS 
REMOVE  OR  REPLACE  MINOR  HARDWARE  ON  LRU.  TEST  SETS  OR 
MOCKUPS,  SUCH  AS  KNOBS  OR  LAMPS 
SECURE  OR  SAFETY  LRU  IN  AIRCRAFT 

ISOLATE  MALFUNCTIONS  TO  ASSOCIATED  AIRCRAFT  EQUIPMENT 
ISOLATE  MALFUNCTIONS  TO  RELAYS 

ISOLATE  MALFUNCTIONS  TO  WIRING  PROBLEMS  ON  AIRCRAFT 

PERFORM  PREVENTIVE  MAINTENANCE  ON  INERTIAL  AND  DOPPLER 
NAVIGATIONAL  SYSTEMS 
REMOVE  OR  REPLACE  RELAYS 

SOLDER  AVIONIC  INERTIAL  OR  RADAP  NAVIGATION  SYSTEM  WIPING 

SPLICE  AVIONIC  INERTIAL  OR  RADAR  NAVIGATION  SYSTEM  WIRING 

PEMOVE  OR  REPLACE  AVIONIC  INERTIAL  OR  RADAR  NAVIGATION 
SYSTEM  CONNECTORS 
PEMOVE  OR  INSTALL  ACCESS  PANELS 

TAPE  AVIONIC  INERTIAL  OR  RADAR  NAVIGATION  SYSTEM  WIRING 
TAG  OR  LABEL  EQUIPMENT 
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* 


For  each  type  of  training,  please  indicate  the  time  you  believe  is 
currently  devoted  to  reach  MINIMUM  STANDARDS  for  the  group  of  tasks  on 
the  opposite  page. 


Training 

Time 

Estimate 
(in  hrs) 


TYPES  OF  TRAINING 


Classroom  instruction  involving  lecture/discussion 
&  related  reading  (most  resident  technical  training) 

Correspondence  courses;  self-paced,  individual  study 
from  text  (all  CDCs) 

Hands-on  experience  in  small,  supervised  training 
groups  using  simulators,  mock-ups,  or  actual 
equipment  (most  FTDs ) 

Hands-on  experience  on-the-job  including  observing 
others,  practicing  the  tasks,  &  receiving  direction 
(qualification  i  upgrade  training) 

Other,  please  specify: _ 
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INSTRUCTIONS— IDEAL  TRAINING 


For  each  type  of  training  described  in  the  questionnaire,  you  are  to 
indicate  the  time  you  believe  should  be  devoted  to  reach  minimum 
standards  in  the  MOST  EFFECTIVE  WAY.  Making  the  most  effective  use  of 
each  type  of  training  may  involve  providing  more  of  some  types  of 
training  and  less  of  others.  Or  it  may  involve  keeping  the  same  levels 
as  the  current  training  system. 

Note:  Ideal  training  does  NOT  mean  providing  more  of  all  types  of 
training,  since  this  means  you  are  raising  minimum  standards,  not 
reaching  the  same  standards  more  effectively.  Ideal  training  does  mean 
providing  the  best  "MIX"  of  training  (e.g.,  more  of  some  types,  less  of 
others)  to  reach  the  SAME  MINIMUM  STANDARDS  you  described  under  current 
training. 


task  MODULE  2-:  PEPFQPM  AND  DOCUMENT  NAVIGATION  EQUIPMENT  MAINTENANCE  CN  AIPCPAFT 

LOCATE  PARTS  OR  STOCK  NUMBERS  IN  TECHNICAL  PUBLICATIONS 

RESEARCH  OR  IDENTIFY  PARTS  USING  ILLUSTRATED  PAPTS 
BREAKDOWNS  (  IPB  ) 

MAKE  ENTRIES  ON  AFTO  FORMS  3A9  (MAINTENANCE  DATA  COLLECTION 
RECOP  D ) 

MAKE  ENTRIES  ON  AFTO  FORMS  2S0  (REPARABLE  ITEM  PROCESSING 
TAG) 

OPERATE  INERTIAL  AND  RADAR  NAVAGATIONAL  EQUIPMENT  FOR 
CHECK-OUT  OF  ASSOCIATED  AVIONIC  SYSTEMS 
REMOVE  OR  REPLACE  MINOR  HARDWAPE  ON  LRU.  TEST  SETS  OR 
MOCKUPS.  SUCH  AS  KNOBS  OR  LAMPS 
SECURE  OR  SAFETY  LRU  IN  AIRCRAFT 

ISOLATE  MALFUNCTIONS  TO  ASSOCIATED  AIRCRAFT  EQUIPMENT 
ISOLATE  MALFUNCTIONS  TO  RELAYS 

ISOLATE  MALFUNCTIONS  TO  WIRING  PROBLEMS  ON  AIRCRAFT 

PERFORM  PREVENTIVE  MAINTENANCE  ON  INERTIAL  AND  DOPPLER 
NAVIGATIONAL  SYSTEMS 
REMOVE  OR  REPLACE  RELAYS 

SOLDER  AVIONIC  INERTIAL  OR  RADAP  NAVIGATION  SYSTEM  WIRING 

SPLICE  AVIONIC  INERTIAL  OR  RADAR  NAVIGATION  SYSTEM  WIRING 

REMOVE  OR  REPLACE  AVIONIC  INERTIAL  OR  RADAR  NAVIGATION 
SYSTEM  CONNECTORS 
REMOVE  OR  INSTALL  ACCESS  PANELS 

TAPE  AVIONIC  INERTIAL  OR  RADAR  NAVIGATION  SYSTEM  WIRING 
TAG  OR  LABEL  EQUIPMENT 
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For  each  type  of  training,  please  indicate  the  time  you  believe  should 
be  devoted  to  reach  minimum  standards  in  the  MOST  EFFECTIVE  WAY  for  the 
group  of  tasks  on  the  opposite  page.  Do  NOT  specify  more  of  all  types 
of  training. 


Training 

Time 

Estimate 
(in  hrs) 


TYPE  OF  TRAINING 

Classroom  instruction  involving  lecture/discussion 
&  related  reading 

Correspondence  courses;  self-paced,  individual  study 
from  text 

Hands-on  experience  In  small,  supervised  training 
groups  using  simualtors,  mock-ups,  or  actual 
equi pment 

Hands-on  experience  on-the-job  including  observing 
others,  practicing  the  tasks,  S  receiving  d’  ection 

Other,  please  specify: _ 
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INSTRUCTIONS— MAXIMUM  TRAINING  OF  EACH  TYPE 


It  may  be  possible  to  completely  train  a  group  of  tasks  using  some  types 
of  training,  but  not  others.  For  example,  a  group  of  tasks  which 
involved  manual  skills  probably  can  NOT  be  trained  to  minimum  standards 
in  a  classroom.  Similarly,  it  might  NOT  be  acceptable  to  completely 
train  some  tasks  on-the-job  without  some  classroom  instructions  on  theory 
or  safety  precautions. 

For  each  type  of  training  described  in  the  questionnaire,  you  are  to 
indicate  the  time  you  believe  that  would  be  required  to  COMPLETELY  train 
each  group  of  tasks  to  MINIMUM  STANDARDS.  If  the  tasks  cannot  be 
completely  trained  using  a  particular  type  of  training,  indicate  the 
PERCENTAGE  of  full  training  that  can  be  provided  and  the  time  it  would 
take  to  reach  this  level. 

The  example  on  the  next  page  shows  you  what  a  completed  questionnaire 
woul d  look  1  ike. 
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EXAWLE 


TASKS: 

Type  letters  from  rough  draft. 

Type  reports  from  rough  draft. 

For  each  type  of  training,  please  indicate  the  time  you  believe  it  would 
take  to  completely  train  the  tasks  above  to  minimum  standards.  If  the 
tasks  cannot  be  completely  trained  using  a  particular  type  of  training, 


please  indicate  the  percentage  of  full  training  that  can 
the  time  it  would  take  to  reach  this  level. 

Training 

Time 

Estimate 
(in  hrs) 

TYPES  OF  TRAINING 

be  provided  and 

Percent 
of  Full 
Training 

Classroom  instruction  involving  lecture/ 
discussion  A  related  reading 

-3- 

Correspondence  courses;  self-paced, 
individual  study  from  text 

2.5-7. 

Hands-on  experience  in  small,  supervised 
training  groups  using  simulators, 
mock-ups,  or  actual  equipment 

7- 

Hands-on  experience  on-the-job  including 
observing  others,  practicing  the  tasks, 

_2_ 

A  receiving  direction 
Other,  please  specify: 


Please  turn  to  the  next  page  and  continue. 
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S  K  MODULE  2A:  PERFORM  AND  DGCUME  II  T  NAVIGATION  EQUIPMENT  MAINTENANCE  ON  AIPCR 

LOCATE  PARTS  OR  STOCK  NUMBERS  III  TECHNICAL  PUBLICATIONS 

RESEAPCH  OR  IDENTIFY  PARTS  USING  ILLUSTRATED  PARTS 
BREAKDOWNS  (  IPS  ) 

MAKE  ENTRIES  ON  AF  TO  FORMS  3A9  (MAINTENANCE  DATA  COLLECTION 
RECORD) 

MAKE  ENTRIES  ON  AF T 0  FORMS  35  0  (REPARA8LE  ITEM  PROCESSING 
TAG  ) 

operate  inertial  and  radar  navagational  equipment  for 

CHECK-OUT  OF  ASSOCIATED  AVIONIC  SYSTEMS 
REMOVE  OR  REPLACE  MINOR  HARDWARE  ON  LRU,  TEST  SETS  OR 
MOCKUPS.  SUCH  AS  KNOBS  OR  LAMPS 
SECURE  OR  SAFETY  LRU  IN  AIRCRAFT 

ISOLATE  MALFUNCTIONS  TO  ASSOCIATED  AIRCRAFT  EQUIPMENT 
ISOLATE  MALFUNCTIONS  TO  RELAYS 

ISOLATE  MALFUNCTIONS  TO  WIRING  PROBLEMS  ON  AIRCRAFT 

PERFORM  PREVENTIVE  MAINTENANCE  ON  INERTIAL  AND  DOPPLER 
NAVIGATIONAL  SYSTEMS 
REMOVE  OR  REPLACE  RELAYS 

SOLDER  AVIONIC  INERTIAL  OR  RADAP  NAVIGATION  SYSTEM  WIRING 

SPLICE  AVIONIC  INERTIAL  OR  RADAR  NAVIGATION  SYSTEM  WIRING 

REMOVE  OR  REPLACE  AVIONIC  INERTIAL  CR  RADAR  NAVIGATION 
SYSTEM  CONNECTORS 
REMOVE  OR  INSTALL  ACCESS  PANELS 

TAPE  AVIONIC  INERTIAL  OR  RADAR  NAVIGATION  SYSTEM  WIRING 
TAG  OR  LABEL  EQUIPMENT 
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For  each  type  of  training,  please  indicate  the  time  you  believe  it  would 
take  to  completely  train  these  tasks  to  minimum  standards.  If  the  tasks 
cannot  be  completely  trained  using  a  particular  type  of  training,  please 
indicate  the  percentage  of  full  training  that  can  be  provided  and  the 
time  it  would  take  to  reach  this  level. 


Training  Percent 

Time  of  Full 

Estimate  Training 

(in  hrs) 


TYPES  OF  TRAINING 

Classroom  Instruction  involving  lecture/  _  _ % 

discussion  A  related  reading 

Correspondence  courses;  self-paced,  _  _ * 

individual  stucty  from  text 

Hands-on  experience  in  small,  supervised  _  _ % 

training  groups  using  simulators, 
mock-ups,  or  actual  equipment 

Hands-on  experience  on-the-job  Including  _  _ % 

observing  others,  practicing  the  tasks, 

A  receiving  direction 

Other,  please  specify: _  _  _ 
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